Draft Analysis

[grdawg]

I've been analyzing the draft data in excel and I came across something interesting. When you create html for the league files, you get a file called draftpreview which can be opened up in excel and gives you the exact table from the draft screen in the game. When looking at it, I noticed that the grades were different then what was in the game. Looking at the data, it appears that the grades you get in the export are your scout's grades and match up to how they rate the player when you interview them.

HTML In Game
Player Pos Grade Adj Grade Grade Adj Grade
Wesley Winslett WR 7 7.3 6.1 6.4 U
Jason Illes CB 6.9 7.1 7.1 7.3 S
Sammy Lemmon RB 6.5 6.9 7.2 7.7 S
Colin Fredrickson CB 5.8 6 7 7.2 VO
Levon Jackson CB 6.9 7.1 6.3 6.5 U
Tommie Ascencao WR 5.8 6 6 6.3 O
Vince Stone RB 6.2 6.6 5.7 6 S
Jake Flemming WR 6.5 6.8 5.7 5.9 U
Charlie Hawk RB 6.2 6.6 4.6 4.9 VU
Cedric Settelmaier CB 6.4 6.6 6.2 6.4 S
Ben Cassidy WR 5.5 5.7 5 5.2 S
Bryant Dodge RB 4.8 5.1 5.9 6.3 VO
Jim Dougan WR 6 6.3 4.8 5 S
Roderick Osborne WR 5.4 5.6 4.7 4.9 S
Stephen Gillespie RB 5.8 6.2 5.8 6.2 S
Roosevelt Irwin WR 5.2 5.4 4.8 5 S
Pat Earley WR 4.6 4.8 5.4 5.6 VO


The first list of grades is from the html export, the 2nd list is from the game. My scout is good in WR, Avg in RB, and Fair in CB. For the first guy, Wesley Winslett, in the html export, he is rated 7/7.3 where in the game he's rated 6.1/6.4. After interviewing him, my scout says he's underrated which makes sense based on those grades. Now look at Colin Fredrickson, he is rated in the html export as 5.8/6 while in the game he's rated 7/7.2, after the interview he is listed as very overrated by my scout which makes sense.

I assume that the amount of difference is represented by how well your scout is rated in the particular position. Now how this translates to actual players abilities I don't know because I haven't drafted yet, but I thought this was very interesting data.

Lastly, do you think this is a bug and this shouldn't be happening? I would think if this holds true throughout then the interview really becomes moot, you could just compare the 2 numbers and see how they differ to get the idea of what your scout really thinks.

I hope this makes sense, I don't post much here, but I figured this was something the community would be interested in.

[gstelmack]

When you create html for the league files, you get a file called draftpreview which can be opened up in excel and gives you the exact table from the draft screen in the game.

Woohoo! Still wish there was a "Print" on the screen, but this is nice to know.

Interesting analysis. I wonder if the bug is in the HTML export, or in the screen? Sounds like at least one case is your scouts interpretation, the other is the amalgamation of all scouts in the game (what the old "bureau rating" was before). Not sure though, Jim would have to chime in.

[Narcizo]

I'll have to study this as it might hold the key to a long-running "discussion" in the GroupThink thread.

[Narcizo]

This is weird. I always presumed that the grade on the draft preview screen in the game was the league wide rating, but surely it is far more logical that the HTML (which is intended for everyone's use in MP leagues) would have the actual league-wide rating while what you see in-game is your own scout's reading of the player. Thus kicking off the "what does underrated and overrated" argument once again, as what you have said would mean that they are the wrong way round.

[Narcizo]

After further testing I'd say that the underrated and overrated tag probably have very little to do with a comparison of the HTML rating (I'll call it "Draft Board rating") and the in-game rating (my scout's rating). I have a pretty good scout (the advantage of taking a 30-year old scout in SP play where no-one is going to pilch him and having played 15-odd years) and if anything is noticeable it's the fact that the draft board generally rates players a lot higher than my scout does, irrespective of whether my scout says the player is over or underrated. 75% of the time players are rated better by the draft board, 25% of them by my scout. However I haven't really looked at enough numbers to ensure that this is statistically reliable. My scout is far from infallible though as he rated a defensive tackle at 6.4, while the board rated him 7.6. The tackle came out of camp at 48-85. And my scout is supposedly excellent at DLine (still only good at "youth" though. Something I have been hoping would improve). I need to test this further and probably will have to get around to doing a MP League test to compare different scouts' views and track the players' development. But that won't be before I actually have to take part in an MP 2007 draft.

Incidentally I have noticed that, as my scout improves, I've been tending to draft fewer duds and more players who actually meet their draft evaluation.

[3ric]

Was just looking through this page, and two column headers appear to be missing (position, weight), and one column is not there at all (height).
http://www.fof2k4utilities.net/IHOF2/html/draftpreview.html

[Narcizo]

First of all it seems I owe an apology. A closer look at the question shows that there does appear to be some correlation between the interview report and a comparison of HTML and in-game draft ratings. And it seems that the idea that the in-game draft board ratings are your own scouts is a non-starter as I found out when I created a test MP league. Every team I checked had the same draft board rating. So that's me batting 0 for 2 at the moment.

To test a few things out I created a test MP league and simmed to the first draft. I then selected the two scouts who seemed to have the most disparate abilities, settling on Philadelphia and San Francisco's scouts. I choose to interview the top 60 wide receivers in the draft for both teams. Philly's scout is rated Excellent for young talent and Very Good for Receivers while the 49ers' ranks average in both. Clearly, you'd expect the Philadelphia scout to provide better, or more accurate, information in the majority, but necessarily all, of the cases. I think the results are pretty revealing of how the draft is scouted at the moment.

http://img297.imageshack.us/img297/8872/receiversaa8.jpg

I've highlighted rows in which the scouts interviews resulted in different evaluations in yellow. Players with "v" next to their name are ranked much higher (one whole draft grade) in the in-game rating while those with "^" are rated much higher in the HTML rating. These are the players that I was most interested in hoping to note some sort of pattern.

Firstly note that all players marked "^" are evaluated as underrated while those with "v" are overrated. Now look across to the post-draft rating. As far as I can tell, as no booms or busts are revealed immediately after the draft, these ratings are pretty similar to the in-draft scout ratings you'd see in 2004. Match them with the HTML ratings and I think you can see that there is a pretty evident match. My idea is that the HTML ratings correspond to the draft ratings as they would have been in 2004. Bob Hodges, rated at 38-71 immediately after the draft, is, consquently, graded higher by the HTML than Blaine Robertson, rated 40-65. So when a scout says that a player is underrated he's comparing his evaluation of the players potential ability with the players position in the draft.

So why are the in-game ratings different? Well, apparently the difference between 2004 and 2007 is that combine results and development influence the in-game draft grade a lot more. So, for whatever reason, Hodges combine result or development bumps him lower down than Blaine Robertson. Which is odd on the face of it as Hodges has better combines across the board. (BJump is included because the format got knackered translating to excel). I think the disparity must partly be down to development and partly to an x-factor, applied in game. Wynn's bump from a lowly 2.7 to a lofty 5.0 is easier to see in terms of combine results though.

A look at the players' post training camp shows what a lot of people have come to expect from the draft. The vast majority of major busts (5 potential or more) occur among players who were evaluated as underrated. All players considered undervalued by both scouts suffers a bust. Meanwhile all players who gained in potential were considered overrated. (all apart from one, Mitch Dennis who was "As Scouted" and who, consequently, dropped back down again).

What seems to happen is that a player has two potentials scored. One "true" potential (the level they will reach if they are given all the playing time they need, and which is hidden) and one "scouted" potential (the potential ability that all scouts work from. How good a scout is will help them get a more exact on the "scouted" potential, but as far as I can see it doesn't help find the "true" potential. This has a rather profound effect on the game.

1. Potential creepers (players with "true" potential higher than "scouted" potential eg. Buddy Wynn) are ignored by the AI because they have a rubbish potential. They wind up being cut and disappear from the game. Meanwhile players that have a scouted potential in excess of their true potential (Rico Humphrey or Chester Flynn to pick one among millions) get to stay in the game even though their true potential is probably lower than the creepers'. This results in a loss of talent from the league, because the AI is unable to perceive true talent.

2. The superior Philadelphia scout actually provides more inaccurate data than the San Francisco scout. Take Blaine Robertson. The Philadelphia scout sees that he will come out of the draft rated around 40-65 and, consequently, believes him to be undervalued for his position in the draft. Meanwhile the 49er scout's reading of Robertson is less accurate, he doesn't see the bars in quite the same place and he views him as being "as scouted". Of course, both are wrong, Robertson in overvalued in the draft. But at least the SF scout is being less hoodwinked by pretty scouting figures than the Philly one. The same holds true for each players there was a discrepency for. The Philly scout is consistently more incorrect than the SF scout.

So how do you go about getting the best players available in the draft? A problem with a lot of boomers is that their current ability is so low that you'll never be able to invest in them enough to get it to start pushing the potential upwards. You need to be able to discern the Roosevelt Bakers and the Dwight Colliers of the draft, players who actually deliver what you expect. It seems that players who are rated "As Scouted" are the best bet of actually delivering what they promise. They don't have the bust-potential of undervalued players while they don't have the woeful starting positions of the combine warriors.

[QuikSand]

Very good work, Narcizo. The correlation with the HTML//In-Game ratings seems very strong to me -- not sure what to make of that, really, though.

On your list of players - the guy Mitch Dennis seems like an odd case. Came out of his first camp with a bump, at least made a roster for the next few years, but he saw his potential rating actually decline after that. I think a player like that would have been impossible in FOF 2004, absent an injury or position switch.

[JeffW]

On your list of players - the guy Mitch Dennis seems like an odd case. Came out of his first camp with a bump, at least made a roster for the next few years, but he saw his potential rating actually decline after that. I think a player like that would have been impossible in FOF 2004, absent an injury or position switch.

Yep, I think that also happened with an undrafted CB with terrible combines in the GT dynasty--Hugh Knight I thnk.

[RedKingGold]

Could it be possible that the scout's ratings are related to under-rated/over-rated for current value and not future value?

[RedKingGold]

Dola, I'm leaning towards the idea that the scout's impression only gives you an idea of his current value; whereas combine numbers give you an impression of whether the player is likely to reach the higher potential value.

[albionmoonlight]

Great work, Narcizo.

[Warhammer]

I'm not sure how much this really sheds light on the situation. Most of the players appear to have had their ratings hammered, or at least decrease significantly over time.

[MalcPow]

I'm not sure what conclusions we can really draw from any of this, but I think we're seeing a few things with your study (and great work by the way). The first is this, when you generate html in an SP game the rating that shows up in the html's draft preview is your scout's view of the player, if you compare that with the league-wide average that shows up in the draft screen and it's higher, your scout will say the player is underrated. This view also seems to hold immediately following the draft, and that's why we're seeing such a strong correlation between the html number and the initial post-draft rating. I'm assuming you simply looked all these guys up post draft while controlling the same mp team as you ran the html report from (conceivably not philly or sf)? I'm going to also assume that you didn't run separate html reports when you decided to take control of philly and sf for the interviewing? So the html number we're seeing is a third scout's view, as are the post-draft and later assessments?

I make these guesses because I think it helps explain a player like Freddie Sanford, who both teams see as Very Overrated (and who time will show is) but the html shows as basically as scouted or a bit underrated. I also mention it because we're trying to judge the effectiveness of these other two scouts but we're really using a third scout (who's also flawed, presumably more than philly but less than sf) as the benchmark to judge them against because that's where these later ratings are coming from. So I think we're reaching to draw conclusions about whether philly's scout, despite the higher scouting ratings, is "wrong" more, when it's just as likely that the sf scout only looks right because we're comparing his assessments with later assessments that are similarly poor.

The one thing that we could take away from all this is that it's now possible that Jim has adjusted scouting error in such a way that "creepers" are not so obvious. In the past we saw consistent growth in ratings over time, and maybe that's still going on beneath the hood, but scouts no longer give such straight line progressive assessments. Who knows...

What are the ratings for the scout you used to get the post-draft and later player ratings, and presumably whose eyes the html was generated from? That might help us figure out some more things.

[Samdari]

I'm not sure what conclusions we can really draw from any of this, but I think we're seeing a few things with your study (and great work by the way). The first is this, when you generate html in an SP game the rating that shows up in the html's draft preview is your scout's view of the player, if you compare that with the league-wide average that shows up in the draft screen and it's higher

I did a test MP league where the commish file and 5 teams were set up on my machine. Both the in game ratings and the HTML export ratings were identical for all 5 teams (but HTML and in game were different). And I had chosen those 5 teams based on trying to get a good mix of scout skills.

I am not sure where the difference comes from, but it does not appear that either of the two ratings sets vary from scout to scout.

[MalcPow]

I did a test MP league where the commish file and 5 teams were set up on my machine. Both the in game ratings and the HTML export ratings were identical for all 5 teams (but HTML and in game were different). And I had chosen those 5 teams based on trying to get a good mix of scout skills.

I am not sure where the difference comes from, but it does not appear that either of the two ratings sets vary from scout to scout.

Interesting. In the looking that I've done with SP careers the html discrepancies vs. the draft screen league average rating seem to match very closely with my scout's under or overrated assessments, leading me to believe that the html was presenting his view. It's possible that mp leagues are set to generate the same html (and so one single scout's personal assessment is shown no matter which team you're controlling when you generate) so that leagues with multiple commishes aren't generating different html. Although I'm guessing Jim never intended the html to generate different figures from the league averages shown on the draft screen anyway.

I still think there's a problem in the way we're looking at the post-draft and beyond ratings regardless though. If you still have the file Narcizo can you take a quick look at any discrepancies between the way the philly or sf scout views a few of these players post-draft, camp, or etc?

[adubroff]

First off...wow that's really neat to see all laid out.

Second, I think one thing we're seeing here is that a lot of these players bust. I wonder how much of that "busting" is due to the AI team where they land not playing them. I'd be suprised (even in this version, where athletics are not as strong a predictor of success as they were in '04) if Buddy Wynn stayed a below average player if he got playing time, but maybe I'm still adjusting to 07....

I've seen some positive breakouts in my own careers....One interesting thing I've seen is that I think the players themselves know when they have potential. I have an RB that I picked up as an UDFA, who came in at like 28/31. He was the worst of my three rbs, and he didn't get slotted for any pt. Around game 8, he showed up on my locker room report as being upset. Given that I was going nowhere and not particularly enthused with my RB, I gave him the final 4 games ...the next two years he showed a 4 pt increase...

[grdawg]

I was the original poster and I have been doing extensive analysis simliar to what Narcizo has done. I would be interested to here Jim's opinion because it seems that the html is the "true" rating, what they'll be rated after the draft, but the in game number is what the other teams go by for drafting purposes. (was this Jim's intention or it the html export pulling the wrong number?) I usually see the higher rated in game player drafted by the other teams. The problem I am having is finding those hidden gems. I keep a spreadsheet from each draft and then compare the html and in game ratings for usually 60 players (the amt I can interview). I also use Skydog's idea of the AAA player that he wrote about a few years ago about 2004. I see how many standard deviations a player is for each of his combine scores then use a weighted average to come up with a combine number, I then add that number to their Adj draft position and usually go by that number for drafting purposes. Usually I can pick out the guy that will have the highest future potential after the draft, however, I am having a hard time developing these players that aren't for sure stars. I also haven't come up with a way to factor their development into their draft number, other then just eyeballing it when drafting.

I try to go back and look to see how a random sample of player's from each draft's career goes, but since I spend so much time analyzing each draft, I am only about 5 years into this career so in 10-15 I should have a better idea to see if anything jumps out, but its hard because not all players will get the playing time they need to reach their potential. Hopefully something stands out from the data.

If anyone is interested in my spreadsheets, let me know.

[adubroff]

I also use Skydog's idea of the AAA player that he wrote about a few years ago about 2004. I see how many standard deviations a player is for each of his combine scores then use a weighted average to come up with a combine number, I then add that number to their Adj draft position and usually go by that number for drafting purposes. Usually I can pick out the guy that will have the highest future potential after the draft, however, I am having a hard time developing these players that aren't for sure stars. I also haven't come up with a way to factor their development into their draft number, other then just eyeballing it when drafting.


I've used that in the previous games and have tried to use it here as well and it's pretty good for the first few rounds....but in this version, the good athlete doesn't last that long, whereas in previous years they would last (particularly some positions).

One thing I've been messing around with is looking for bars which "don't make sense" ...as an example, I'll sort say tackles by strength...I'll look at each ones blocking strength and see if there's a guy who's bar looks suspiciously low(or high) next to guys with similar strength. I figure this guy is likely to show positive development over time. The one thing with it is that a lot of these guys show initial potential in the <30 range so even if they break through, they might only do so to the high 30 (decent backup) level. It's also pretty tedious to do this, you end up scanning through fields of 60 guys 2/3 times each and it takes a bit to decide whether what you are looking at is what you wanna see...

[Narcizo]

Usually I can pick out the guy that will have the highest future potential after the draft, however, I am having a hard time developing these players that aren't for sure stars. I also haven't come up with a way to factor their development into their draft number, other then just eyeballing it when drafting.


This is what I was alluding to when I said taking the combine warriors often doesn't pay off, even if you "know" that their potential ability is a lot greater than you're being told by your scout. To make it pay off you have to put a lot of faith in a player who looks crap. (and it seems that these combine guys have been made (to look?) much worse in 2007). What if it turns out that you were wrong. They get the odd bump in potential in camp and after the first year, but after playing them for years they level out only about 10 above where they were when you picked them. You'd be better off picking a certain bust who at least has the current ability to play. Trying to find a balancing act here is what makes the draft in 2007 much more challenging than the simple "draft by numbers" approach you could use in 2004.

[Narcizo]

I'm not sure what conclusions we can really draw from any of this, but I think we're seeing a few things with your study (and great work by the way). The first is this, when you generate html in an SP game the rating that shows up in the html's draft preview is your scout's view of the player, if you compare that with the league-wide average that shows up in the draft screen and it's higher, your scout will say the player is underrated. This view also seems to hold immediately following the draft, and that's why we're seeing such a strong correlation between the html number and the initial post-draft rating. I'm assuming you simply looked all these guys up post draft while controlling the same mp team as you ran the html report from (conceivably not philly or sf)? I'm going to also assume that you didn't run separate html reports when you decided to take control of philly and sf for the interviewing? So the html number we're seeing is a third scout's view, as are the post-draft and later assessments?

I make these guesses because I think it helps explain a player like Freddie Sanford, who both teams see as Very Overrated (and who time will show is) but the html shows as basically as scouted or a bit underrated. I also mention it because we're trying to judge the effectiveness of these other two scouts but we're really using a third scout (who's also flawed, presumably more than philly but less than sf) as the benchmark to judge them against because that's where these later ratings are coming from. So I think we're reaching to draw conclusions about whether philly's scout, despite the higher scouting ratings, is "wrong" more, when it's just as likely that the sf scout only looks right because we're comparing his assessments with later assessments that are similarly poor.

The one thing that we could take away from all this is that it's now possible that Jim has adjusted scouting error in such a way that "creepers" are not so obvious. In the past we saw consistent growth in ratings over time, and maybe that's still going on beneath the hood, but scouts no longer give such straight line progressive assessments. Who knows...

What are the ratings for the scout you used to get the post-draft and later player ratings, and presumably whose eyes the html was generated from? That might help us figure out some more things.

I looked at the players using the Philly and SF scouts. The figures I've given are from the Philly's scouts but, oddly, I didn't find there to be a great deal of variance between the two scout's evaluation. Maybe a point or two here or there. Nothing I could be bothered tracking seperately, I'm afraid.

FWIW I think Jim has alluded to scouting error being greater in this version, which, I suspect means, that the combine warriors are better than they look even after they're drafted and are progressing nicely - you just don't know it.

I think that what we have learned from this is
a) Interviews aren't really very satisfactory at the moment, as the underrated and overrated tag are, often - although not always, counter-intuitive. And there was me getting excited whenever I found a very underrated player when the game first came out. :( However, as I said already, in many ways he draft is far more interesting, challenging and satisfying than the draft in 2004, and with a few tweaks should be fantastic.
b) The AI needs to be given access to more information so it can make better decisions about which players to cut or keep. Ironically the AI performed best in this respect under the "a" patch. That's because the problem with current rating scouting error often revealed a players' "true" potential, meaning that the AI placed a lot more value on the creepers (because they weren't creeping anymore, they'd already got there in the AI eyes).
c) Look out for those "as scouted" guys. Apart from the ones who best anyway. :D

[Ksyrup]

I'm enjoying the draft far more than I thought I would. My last draft, I was able to pick 7 rounds of draftees with none of them having less than a '50' future, which is what I look for. The best was a 71 future WR taken at 2(32) and a 69 future NT taken at 1(32). After training camp, every one of them had added 1-3 points to their current ratings and dropped anywhere from 3-12 points in their potentials. I'll be interested to see if any of that comes back. I was particularly disappointed with the QB I drafted in the 7th round (last pick of the draft) who was 5/60 and was 6/48 after camp. I don't think I've been this addicted to an FOF game since 2K1.