FOF4: Cracking the year end file format

[primelord]

There has been a lot of talk about the ability to keep track of all the stats for every player that has come through the FOF universe. While the game doesn't track this information once a player retires unless they go into the hall of fame that information could be available to us. The files that are generated after every season contain the game stats for every player of that year. If we can crack the file format of the 00000001.1997 etc files a program could be written to read that data and generate all kinds of reports. Things like teh ability to serach for stats on any player that has ever played, yearly league leaders etc etc.

I have been working on this for a little while, but I am stuck so I figured I would bring it to the board. There are several people here who are much better at cracking these file formats than I am. So feel free to join in and help crack the code.

[primelord]

Here is a quick run down of what I have found so far. I have figured out which fields hold the game data for every player and what those fields are. For example I know which fields when I hit a plyer record are pass completions, rushing attempts etc.

The biggest problem I have had so far is not being able to determine which records belong to which players. The game doesn't seem to link the game data to the players name. I assume it is linked to some player ID the game uses internally, but I have not been able to figure out which field that is or if it is possible to even figure out which player belongs to which ID. I need to get my notes together, but I will post which fields hold the stats in a bit.

[3ric]

Sounds interesting. The FOF4 roster files begin like this:
id number - jersey number - length of first name - first name - length of last name - last name ... and so on.

I'll take a look at your data to see if I recognize anything.

Edit: forgot a data field

[Fido]

Originally posted by primelord
The biggest problem I have had so far is not being able to determine which records belong to which players. The game doesn't seem to link the game data to the players name. I assume it is linked to some player ID the game uses internally, but I have not been able to figure out which field that is or if it is possible to even figure out which player belongs to which ID. I need to get my notes together, but I will post which fields hold the stats in a bit.

Don't own FOF4, so the best I can do is give some ideas where to look. You might check a one to one comparison (does the first player in teh databse match up with the first year end stats?). You could also try checking the player records to see if they contain an index into the year end stats.

[primelord]

Thanks for the input guys. You two were the ones that came right to mind when I said there were several people better at this than me. I know there are a few other guys who are good at this stuff too.

[Aylmar]

Hmm...well, if the first field in the roster file is the player ID, then see if there is a similar looking column in the year-end file that will correspond with it. It's possible that the year-end stats are simply keyed by player ID (in the first field of the row, I mean). If a different key was used for the stats, it's still likely that you'll be able to locate the player ID field somewhere in the year-end statistics file, since it is player based. Once you think you've found it...a simple comparison to last year's statistics (with a subset of players, of course), should tell the tale.

[primelord]

Aylmar,

Thats a good idea and that may be a very good starting point. However wherew e may run into a problem is trying to determine player ID's for players generated within FOF4. Because these players aren't in the roster file and there is no draft file for them either. We would have to read the save game data, but it's possible that information could be found in the save game data too.

[Aylmar]

primelord,

I'm hampered by the fact that I don't own FOF4, but I would think that there has to be a set of files somewhere that associates current players to their names, stats, etc. It makes sense (at least to me), that you'd want a master file for the "universe" with all current players associated to their names (and other static information) and some kind of key that also associates them with the stats that are being generated for that particular season. The trick is finding the file(s) where the active players are stored. Once those are located, then you could probably use the IDs to associate back to year-end stats.

The hitch in all this is that it's possible that the ID-Name association for a player is only being kept around in the data for as long as they are active in the league (this complicates the entry of fictional players into the HOF, but reduces the footprint required by the data files). If that's the case, then mining the sort of data you're talking about could very well be enormously complex, if it's even possible.

Edit -- In rereading what you wrote, I would think that the save game files would be a very good place to look for the files that I'm talking about. That's what I get for not paying attention, I guess.

[primelord]

It looks it will have to be tonight before I get my findings posted. The save game file could have promise. The names of players in FOF4 are hard coded in the save game files. So if one of the fileds right after a players name is the player ID, it may end up being easy figuring out which one he is in the year end file.

[primelord]

Ok here is what I have so far. Currently I kind of brute force things to find a player in the year end file I am looking for. In this example we will be using Otis Coghill who is our starting QB in the group think project. I load up FOF4 and see that in 1996 in his first game Coghill threw for 258 yards. So I load the 1996 year end file and do a search for the hex value of 258. In this case it is 0201.

BTW quick side note. If you are trying to edit hex files for the first time here is something that completely threw me for a long time. If you are using a hex converter to find the hex value of a decimal you have to reverse first half and second half of the numbers. For example if you are looking for the value 258 the converter will tell you the value is 0102. When you search the file for that value you have to serach for 0201. I don't know why that is, but it took me awhile to realize that.

So anyway to figure out which record is Coghills ijust up every 258 I come acroos to 259 until I find my guy. I find the first record for Coghill and here is what I have found out so far.

Unknown
Year
Team Number
Game Played
Game Started
Attempts
Completions
Yards Passing
Longest Completion
TD Passes
Ints
Sacks
Sack Yards
Rushes
Rushing Yards
Longest Run
Rushing TD's
Catches
Receiving Yards
Longest Reception
TD Catches
Targeted
Yards After Catch
Drops
Punt Returns
Punt Return Yards
Punt Return TD's
Kick Returns
Kick Return Yards
Kick Return TD's
Fumbles
Fumble Recoveries
Forced Fumbles
Total TD's
Key Run Blocks
Key Run Block Opportunities
Sacks Allowed
Tackles
Assists
Sacks
Ints
Int Return Yards
Ints Returned for TD
Passes Defensed
Passes Blocked
Hurries
Catches Given Up
Pass Plays
Run Plays
Field Goals
Field Goal Attempts
Longest Field Goal
PAT's
PAT Attempts
Punts
Punt Yards
Longest Punt
Punts Inside 20
Total Points

And then the it starts over again for the next game.

[primelord]

Now my guess would be maybe that Unknown field at the start of the record may be how we distinguish the player, but its a different value before each game. For Coghills first game that field was 7. For his second game it was 12. And here is the order for all the games he played.

7
12
5
14
22
30
28
6
2
11
12
10
20
9
28
15
11

[primelord]

I am wondering if that unknown field might be the team number the opposing team. When I alter that field it doesn't seem to change anything on the players stat acreen including the name of the opponent so I am not sure.

[henry296]

3 numbers repeat...

11, 12, 28. Are those games agaist division opponents?

Todd

[primelord]

Originally posted by henry296
3 numbers repeat...

11, 12, 28. Are those games agaist division opponents?

Todd

Yea those numbers all correspond to the team numbers for our division rivals so I guess that settles that. Those have to be the team numbers for the opposition.

I am still confused on how the game knows who is who. Although I am guessing it is possible the file may just be split up into 17 week sections and the first 17 sections belong to the player with ID 1, and the second 17 belong to player ID 2 etc...

[primelord]

Bump to see if anyone else has any ideas yet.

[cthomer5000]

Originally posted by primelord
Bump to see if anyone else has any ideas yet.

I'm very interested, but i won't be of any help with this file cracking.

[3ric]

What's the introductory values of the file? One of them should be the number of records kept in it. In a draftfile, the hex codes correspond to a beginning string "PlayerDraftFile" and then "04" for the version, and right before the first record, a "945", which is how many available draftees there are in the file.

[sabotai]

I wonder if the 'index' to figure out which record belongs to each player is in another file. Just a thought.

[Anrhydeddu]

Since you have a direct pipeline to Jim, when not ask him?

[3ric]

When you found Coghill's stats in the file, were they adjacent to each other or spaced out with intervals? If the latter, then the intervals should be the same (in bytes), if the 17 weekly sections theory holds.

[primelord]

In the year end file it starts with the text block "PSeasoDataFile" then "04". The next field in the file is a rather large number. In the 1996 file it is 13088. Then it looks like the first player record starts. So it seems reasonable that that number represents the number of game records in the file. There is no way there are 13,000+ players in the file, but total number of game stats that seems like it would be about right.

That still confuses me then on how it knows which records belong to which player.

[primelord]

Originally posted by Anrhydeddu
Since you have a direct pipeline to Jim, when not ask him?

Heh I did.

[primelord]

Originally posted by 3ric
When you found Coghill's stats in the file, were they adjacent to each other or spaced out with intervals? If the latter, then the intervals should be the same (in bytes), if the 17 weekly sections theory holds.

All the player records run right into each other. So the field right before Coghills first field is the total points for week 17 of the player behind him. That player is on a different team so they aren't grouped together by team either. Then Coghill has 17 sets in the file before the next player starts.

[3ric]

13088 divided by 16 is 818, which is a too small number to be all players on all teams plus free agents, which is a little strange...

[primelord]

Originally posted by 3ric
Nevermind...

I think it would be by 17 though. It looks like each player has a record for all 17 weeks. And they all just have no stats for the week they had a bye.

Edit: heh you changed it as I was quoting it.

[3ric]

Sorry about that, I realized it didn't add together, so I wanted to re-do it.

I have opened another year-end file, this one with 14640 records. Divided by 16 it's 915. I don't think it's a coincidence...

[primelord]

The beginning part of the file contains a lot of records of people on team 99. Which should mean they are a free agent, but it looks like there are players who are mixed in that are free agents too.

[3ric]

Originally posted by primelord
7
12
5
14
22
30
28
6
2
11
12
10
20
9
28
15
11

You provided the answer yourself, since Coghill appears 17 times here then there should be 17 records for each player. But which one of these is the bye week?

[primelord]

Originally posted by 3ric
Sorry about that, I realized it didn't add together, so I wanted to re-do it.

I have opened another year-end file, this one with 14640 records. Divided by 16 it's 915. I don't think it's a coincidence...

Yea I think you are right. I opened up 3 other year end files and they all divided evenly by 16 too. I wonder why Coghill had 17 entries then. I guess I should recheck that and make sure he does.

[primelord]

Originally posted by 3ric
You provided the answer yourself, since Coghill appears 17 times here then there should be 17 records for each player. But which one of these is the bye week?

Well Coghill may not even be thebest example because he was hurt a couple games so he has multiple entries without any stats. Let me find a guy who played all 16 games.

The weird thing though is that all those numbers should correspond to a valid team. I would have thought the by week would have like team 99 in it or something.

[3ric]

I'm thinking there should be some extra bytes to a record we don't know about yet. My file is 5594146 bytes big after subtracting the initial 18 bytes, divided with the number of records (14640) gives 382,113797814207...

That don't match with the theory that a record is about 118 bytes long.

As for the order the players are arranged in, that has to be that they are sorted by index number, even though it isn't listed.

The only way to find out could be to give the players index numbers starting from 1 for the first player in the file and his 16 or 17 weeks. Then find a particular player in the *.fga file and compare his index there with the one attributed to him based on his position in the file.

[primelord]

Ok so now I am looking at our starting RB Jamal Lucas who started all 16 games for us in 1996. Ok well Lucas only appears to have 16 entries so now I need to go back and count Cog's maybe I made a mistake.

[VPI97]

Originally posted by primelord
I wonder why Coghill had 17 entries then.

Playoffs?

[primelord]

Originally posted by VPI97
Playoffs?

See I thought that might be it too, and it seemed like it might fit because We onyl had one playoff game that season. But Lucas should have had 17 games too then. And it still just seems like it would be too much of a coincedence for that first number to always divide evenly into 16 if it also took playoffs into account.

[VPI97]

Note - If you guys can find a link from the year-end file to the roster file, I still have the code from the FOF Almanac....that's still a useful front end regardless of the origination of the stats.

[3ric]

Originally posted by 3ric
I'm thinking there should be some extra bytes to a record we don't know about yet. My file is 5594146 bytes big after subtracting the initial 18 bytes, divided with the number of records (14640) gives 382,113797814207...

Now I've got it right. You should subtract 20 bytes from the file's size, and divide it with 16 and 118 to get the number of players in that file, which is a nice even 2963.

53 players*32 teams is 1696 players (not including injured reserve, of course) which would mean 1267 free agents.

There are 18 initial bytes in the file, and I think there's 2 extra bytes at the end of the file that doesn't belong to a record either, but I have no idea if its value is significant.

[3ric]

Dola.
prime, I already have the struct format for the *.fga file available in an Excel file, so you don't need to "crack" it to be able to read the player names from it. And no, I didn't do it myself - another FOFC member emailed it to me a couple of months ago.

[primelord]

Originally posted by 3ric
Dola.
prime, I already have the struct format for the *.fga file available in an Excel file, so you don't need to "crack" it to be able to read the player names from it. And no, I didn't do it myself - another FOFC member emailed it to me a couple of months ago.

Thats excellent. Is it something you can post here or send to me?

[3ric]

http://m1.311.telia.com/~u31120783/FOF4Data.xls

[primelord]

Wow that spreadsheet is good stuff!

Looking at their data for the year end file it looks like the opponents team ID is actually the last record in the record for the game instead of the first. So I am thinking then that all 20 extra bytes are right in the beginning of the file. There is a 2byte short right after the initial 18 bytes that has a value of 0 which wouldn't correspond to a team ID. That was confusing me, but now that I see the opposing team ID is the end of the record instead of the beginning I think it is just the extra two bytes you were talking about.

[primelord]

Heh that file has all the information we need right there. In the player records in the save game files there are fields that contain the offset for the player in the year end file.

[vulcan]

*BUMP*

[vulcan]

....and how far are you guys into this? Man, this would be great. I have been gone for a long while but I still come around...and NO I didn't die like crushem thought I did.