Trying to understand Draft Analyzer better

[wishbone]

Hello,
Just trying to understand the Draft Analyzer Utility better and looking for input on the below.
I see players listed like this linebacker:
http://i1054.photobucket.com/albums/s500/dayton52/davenport1.jpg (http://s1054.photobucket.com/user/dayton52/media/davenport1.jpg.html)

My understanding is that Combines and Rating columns are based on the weights I enter. These look like:
http://i1054.photobucket.com/albums/s500/dayton52/davenport2.jpg (http://s1054.photobucket.com/user/dayton52/media/davenport2.jpg.html)

What I don't get is The D? VG G A and FP Columns. These seem to indicate that he will most likely develop to 40+ and has a 62% chance of being Good or Very Good.

I feel like I don't understand the tool so any help is appreciated.
Do I need to change the weights dramatically?
I feel like Draft Analyzer has helped me not just look for big bars or good combine scores when drafting and with the ability to easily see threshold numbers I feel like the average talent level on my team is going up but feel like I am using it wrong.

I took a flyer on the LB above, here he is after Training Camp:
http://i1054.photobucket.com/albums/s500/dayton52/davenport3.jpg (http://s1054.photobucket.com/user/dayton52/media/davenport3.jpg.html)

[gstelmack]

There are 2 different approaches to drafting available in the utility:

1) The original method - weights for combines and bars that add up to a score. In this case, the player has 1 combine that is below the "threshold" for the position mentioned in Ben's thread (What are the important combines for each position group? - Front Office Football Central (http://www.operationsports.com/fofc/showthread.php?t=89290)) that means he has a low chance of turning out >50, which is a -100 penalty according to the default weights setup, and he has a really bad 40 time, so gets a very low combine weight.

2) The new machine learning method, where I'm using machine learning algorithms to classify players. This is still in flux, we don't know yet how accurate it will turn out to be, and I'm working to improve it. But yes, he has an 89% chance of turning out 40+ (and an 11% chance of being below 40), a 62% chance of being Good or better (51+), a 34% chance of being Average (31-50), and a 2% chance of being a total dweeb-ball.

Note that he has a wide spread under the "new" system. That's an indication that the routine really isn't quite sure what to make of him. Look at the other guys below him and you'll see a much narrower spread in those columns.

Note that his bars match those combines - very low pass rush technique, strength, and punishing hitter. The question is, how big will the other bars get? It is very likely the low bars are accurate, but if he grows in the others (especially run defense) he well could turn into a high-40s / low-50s LB that can't rush the passer worth a darn but can stop the run and maybe play a little coverage. Note that the default weights (and FOF, since that's where they are from) don't value those low bars particularly highly for a LB.

Or he won't grow any of them and he'll fall into that 11% D? range and 34% Avg range. But he can still be a reasonable backup LB. The 2nd round slotting might be high here (it may not handle these toss-up guys very well and may need more work), but he was probably worth a swing in the 7th round to provide depth.

[aston217]

How accurate are those percentages that DA quotes? Because in the DA thread, you note two things:

1) Broadly, DA is 50% right about the classifier (VG/G/A/FP) to which it assigns the highest percentage.

2) Broadly, DA is 95% within one classifier of the above prediction.

So, let's take a guy who is 60% G, and a guy who is 42% G. Both are in the highest classification. Is the first one 18% strictly more likely to be good? Or are they roughly equivalently likely to be G/VG/A given what we *know* to be the accuracy of these numbers DA produces? At what point can we say, "The utility knows what to make of this guy" with high confidence?

[wishbone]

Thanks for the insight Greg. Obviously addressing individual players is not efficient and I appreciate you taking the time.

[gstelmack]

How accurate are those percentages that DA quotes? Because in the DA thread, you note two things:

1) Broadly, DA is 50% right about the classifier (VG/G/A/FP) to which it assigns the highest percentage.

2) Broadly, DA is 95% within one classifier of the above prediction.

So, let's take a guy who is 60% G, and a guy who is 42% G. Both are in the highest classification. Is the first one 18% strictly more likely to be good? Or are they roughly equivalently likely to be G/VG/A given what we *know* to be the accuracy of these numbers DA produces? At what point can we say, "The utility knows what to make of this guy" with high confidence?

1 & 2 mean "if you take the classifier with the highest chance, that is the correct one 50% of the time, and within 1 95% of the time". That is an overall summary of the entire population, or more specifically the summary of my entire test population. When you take an individual, the percentages are different, and yeah the first is more likely to be good. Is it 18% more? I don't know, that requires more analysis.

To really know how useful this information is, it needs to go through some draft classes. I'm looking in to maybe putting these numbers into PlayerTracker (which has the same data) so you can look at fully developed individuals and see what DraftAnalyzer would have thought of them, and see if it's anywhere close, or this entire experiment is a miserable failure. I *suspect* that the approach is a good one, but I may either need more sample data to train it on, or a better training routine. I'm doing some deep belief neural network experiments now that are showing some early signs of getting better, but we'll see how they pan out.

[gstelmack]

Thanks for the insight Greg. Obviously addressing individual players is not efficient and I appreciate you taking the time.

To know if this is going to work we HAVE to look at individual cases in addition to the total population to see if it's any good. We'll either find that it doesn't work, find that it does, or figure out the exceptions.

[garion333]

When I see a spread like that across VG/G/A/FP my first thought is interesting. This guy could be anything.

Based on those big fat 0's in PH and PRS tell me I wouldn't have drafted him. If he had shown some life in those areas, even with that terrible Bench score, I would've given more credence to the VG or G percentages.

Going in round 2. That seems like a stretch and definitely sounds like something that Greg will be tweaking. As he said, taking a 7th on him was probably a good choice. If those pass D bars turn out, he would definitely make my team.

[corbes]

For anyone just starting with draft analyzer, my STRONG, STRONG, STRONG recommendation is to:

1. Sort all players by rating
2. Disregard all players with negative ratings, as these players have at least one below-threshold combine score*
3. Amongst the remaining positive players, use the other tools in draft analyzer to examine and sift to your heart's content.

Messing around with below-threshold combine players is like trying to find a needle in a haystack.

[Hammer]

Maybe I am old fashioned but why would anyone want to know a player has an 89% chance of becoming a good player. Doesn't that spoil the fun of the draft? Surely winning isn't that important. Is it going to get to the point that a program is basically going to draft for you? That would be a shame.

It seems to me that it is much harder to make in impact by writing a gameplan in this new version. I understand people may need a bit of help, but it sounds like it is getting to the point where a utility is gifting you an answer?

I have never used the draft utility, but I understood the skill was in setting the utility up to your own values? Now it sounds that will be bypassed with a program which, if all goes well, will tell you how good a player is. Not exactly designing a nuclear warhead for the first time, but I think this could of implications for the fun factor of the game. Admitted I am not a IT type, so perhaps I am in the minority thinking this way..

[Alf]

I have more fun when I win, so any hint to drafting the correct player is more fun to me in the long run.

Edit : this is particularly true in MultiPlayer leagues

[Ben E Lou]

Is it going to get to the point that a program is basically going to draft for you? That's absolutely the stated goal:... I'd still like to work more on Draft Analyzer - I'm busy and want it to draft for me, and I'm not quite there yet.

A big part of the issue is that some GMs understand the draft far better than others and are already picking good players round after round with much greater consistency than even the program can deliver at this point. It also depends on the drafting ability of others in the league. If you have just 1 or 2 guys in your league who are grabbing 60+ guys in round 3 and 4 routinely, the competitive balance can get out of hand in a hurry. If you have no one in your league doing that, then it's not a problem to you.

[gstelmack]

The question is: will we ever get there? I still don't know if this new stuff is good enough, or if I can improve it to get it good enough (and the problem may not be the approach, but my implementation of the approach). But yes, I'd love to level the playing field and have the software catching the same sorts of patterns that the expert drafters pick up on fairly easily.

The other benefit for me is playing around with some of these data mining / machine learning algorithms and packages out there and learn more about them with a concrete example.

[Hammer]

I guess if truth be told, I don't want to be put in a position where players whip me (and others) who draft old school. I enjoy being beaten, as it makes me come back harder and develops the rivalry. But to get beaten by a machine or utility, mmm, not so much.

In chess there are programs that can beat any human. I could play online again the best guy in the world and beat him now in theory, if I had my little program to help me. Kind of sucks the fun out of playing online when you don't know if the guy your playing beat you, or if he was using a program.

I won't use a chess program or a draft utility. As to me it isn't fun to do so. I am not that bothered about winning at a game which is just a leisure activity. From a selfish point of view I hope the utility doesn't rise above the human as It will push me and a few others into being a SP only FOF.

To this point I admire and respect your work Greg, even if it isn't my bag. But personally I hope you don't become so good at what you do that you force people to use it to stay competitive. Just food for thought.

[garion333]

Don't worry, the randomness inherent in the game hammers us all back down to reality.

[Subby]

I think at its best Draft Analyzer is a safety net for players that are newer to mp leagues. For folks who have been drafting for a while, it's really just a glorified spreadsheet - which is fitting since FOF is the spreadsheetiest of games available.

Don't get me wrong, this is one of the best add-ons ever made for FOF, but I also think the returns are diminishing as players become more comfortable with drafting.

[Hammer]

I agree with that entirely Subby. But I think the danger lies in the future, if it comes too good. Making a them and us scenario. Those who play the game as it was intended being at a distinct disadvantage. I have read this is about leveling the playing field, but it could do the exact opposite. I don't really see the fun in being told who the best player is by a program, but I guess that is besides the point.

[gstelmack]

That ship sailed a long time ago - there are already GMs that are VERY good at drafting by perusing the draft class and figuring out patterns. The playing field is not very level right now. Luckily several of those have been sharing their info to help any who will listen get better.

[aston217]

Yeah, I see this as something that levels the playing field.

Like Hammer, I am uneasy that the machine could get too good. But at this point -- and it's hard to make a quantifying statement without numbers -- it doesn't seem that way. A majority of these 800+ man draft classes are always junk. This makes it maybe more obvious (as if it wasn't enough; maybe just go with 400 man classes, Jim?...) and also flags the solid players, but can't make guarantees.

It's nice to be that one guy that remembers numbers like 5.27 and 7.80 and 37 and whatever else, but everyone who doesn't is at a huge disadvantage. And in my experience, no matter how many times this is explained, most people will not remember the numbers (or ignore them). So, maybe it's not so bad.

Or maybe it is. I did review my most recent CyFL draft class in DA, though. I found a number of picks I wouldn't have made if I had seen the DA numbers. If only it had saved me! ;)

[Nemesis]

That's absolutely the stated goal:

A big part of the issue is that some GMs understand the draft far better than others and are already picking good players round after round with much greater consistency than even the program can deliver at this point. It also depends on the drafting ability of others in the league. If you have just 1 or 2 guys in your league who are grabbing 60+ guys in round 3 and 4 routinely, the competitive balance can get out of hand in a hurry. If you have no one in your league doing that, then it's not a problem to you.

Yeah Ben, he generally IS that guy. :)

[Ben E Lou]

Yeah Ben, he generally IS that guy. :)Then he needs to play in tougher leagues. ;)

[Dutch]

I'm slowly starting to formulate a strategy for effective harnessing of the Draft Analyzer. Seems now with FOF7, a split between bars/combines in the early stages should be about 90/20....and slowly moving more towards heavy combine considerations as the draft progresses.

[BighouseBrock]

i just downloaded the draft analyzer, i've never used it before. i have version 1.3.0.0

i do not see the columns D? VG G A FP

how do i get them to show?

[garion333]

Those are gone now as Jim (the dev) killed their efficacy with a patch.

[Julio Riddols]

Yeah, DA kind of got neutered by that, which I am half thankful for. Now I feel like it is most effective as a tool to find particular players later in drafts. You want a guy who can long snap? Sort by the long snapping bar. Want a guy who can play ST? sort by the ST bar, etc.

[PLD3009]

is the latest version at StelmackSoft FOF2007 Utility Suite (http://www.gstelmack.com/UtilitySuite/)?

Thanks

[tucking fypo]

Check the FOF7 Reference Thread here (http://www.operationsports.com/fofc/showthread.php?t=88538)

[PLD3009]

thanks