Defensive Game Plan Testing Problem

[JeffW]

I've been testing out game plans and I found an interesting little problem that is confounding my attempts to correlate various gameplans with on the field success. The variables I'm tracking are:

Pass Yds/Att Comp% Rush Yds/Att Losses

So far I've been unsuccessful at making any real correlations. One thing I've noticed is that often the best teams allow huge yards/carry on rushes--especially undefeated teams. Only 2 teams have gone undefeated in 50 trials:

PYd/At Comp% RYd/At L
5.60 52.5 5.19 0
5.08 52.6 4.85 0


These teams gave up far more yards/rush than their peers. I suspect that the confounding factor is that teams that are more dominant are less often defending against low-YPC carries(such as if they were behind by 20 points and the other team was running the ball up the middle to run the clock out).

Thus, the truly dominant teams are far more often facing high variance outside runs that have low success rates but gain gaggles of yards every so often. Their Y/C is not necessarily indicative of the quality of their rush defense.

It would be nice if FOF had a rudimentary success rate statistic or at least a variance statistic. A 4 yard run on 1st and 10 is more valuable than a 7 yard run on 3rd and 10. Here is an example of a success rate statistic used by Football Outsiders (http://www.footballoutsiders.com/ramblings.php?p=254&cat=11):


* In general, a play counts as a "hit" if it gains 40% of yards on first down, 60% of yards on second down, and 100% of yards on third down.
* If the team is behind by more than a touchdown in the fourth quarter, the benchmarks switch to 50%/65%/100%.
* If the team is ahead by any amount in the fourth quarter, the benchmarks switch to 30%/50%/100%.

[Mestor]

Firstly, the game does in fact have a football outsiders rushing statistic, I forget exactly what its called but it weights rushing yardage gains by down and distance and also cuts out the huge yards on big runs to try and get an accurate representation of your rb's actual ability and your Olines actual ability.

Its towards the top of the same bar that has the career records and statistics and what not.

As for excellent teams giving up gobs of rushing YPC, it might be a factor of teams getting way behind vs them and they just aggressively defend the pass for whole second halves, while the other team is not throwing it downfield every single down.

I would check this by seeing how teams excellent teams rank in the amount of runs they face per season.

Now the fact that teams that are down are still running the ball might be a gameplanning problem, or it might not, as they seem to be having success doing it.

I certainly don't know anything for sure on this though, since i can't check at work, so take what I say with a grain of salt : )

[JeffW]

As for excellent teams giving up gobs of rushing YPC, it might be a factor of teams getting way behind vs them and they just aggressively defend the pass for whole second halves, while the other team is not throwing it downfield every single down.

That is certainly part of it as well.

I would check this by seeing how teams excellent teams rank in the amount of runs they face per season.

I think both 16-0s were right around 1st in carries "allowed."


Now the fact that teams that are down are still running the ball might be a gameplanning problem, or it might not, as they seem to be having success doing it.

I think the confounding factor is more that they're never ahead and therefore never getting grind it out carries

Re: Adjusted line yards. It's not exactly what I'm talking about. Adjusted line yards separates carries of different lengths, but as far as I know, it does not analyze situations(i.e. it still weights a 4 yard gain on 1st/10 as highly as 4 yards on 3rd/10).

[Mestor]

That is certainly part of it as well.



I think both 16-0s were right around 1st in carries "allowed."


hmmm well shows what I know. They get the least amount of runs against them, and cant defend them at all.


I think the confounding factor is more that they're never ahead and therefore never getting grind it out carries

Re: Adjusted line yards. It's not exactly what I'm talking about. Adjusted line yards separates carries of different lengths, but as far as I know, it does not analyze situations(i.e. it still weights a 4 yard gain on 1st/10 as highly as 4 yards on 3rd/10).

Okay, I wasn't 100% clear on what exactly was measured without actually reading the ingame description. I was under the impression it took down and distance under consideration as well.

[JeffW]

More Data:

In another trial I just ran, my team wen 15-1 and allowed 5.30 yards/carry--1.21 yards above league average and .52 yards above the next worst. The top 7 teams in rushing yards allowed/carry all had losing records--several 3-13, etc. 4 of them allowed 514 to 557 carries compared to 468 league average. The other three allowed 481, 467 and 441.

[Subby]

How did you guys rate against the pass? (YPG, YPA, etc...)

[JeffW]

How did you guys rate against the pass? (YPG, YPA, etc...)

TBH, I don't remember which season goes with which datapoint, but generally the 15-1 16-0 teams are around 27th-32nd in passing attempts allowed, top 10 in passing yards/attempt and somewhere in the middle for passing yards/game(I haven't been tracking yards/game or attempts/game unfortunatley, but the trend is unmistakable).

I guess my point in this thread is that dominant teams aren't as good against the pass as they seem(they face a lot obvious passing situations with poor yards/attempt from opposing defenses) nor as bad against the run as they seem. If one did a regression, they'd see that dominance correlates highly with yards/attempt allowed, but I do not believe the correlation is as strong between dominance and actualy quality of pass defense.

[Subby]

It makes sense to me then that the good teams get ahead quickly, defend pass heavy and happily give up 5+ yards per run (and let the clock tick) while playing all out against the pass. That would support what you are finding and kind of mirrors a pretty basic MP defensive philosophy...

[Passacaglia]

Having barely been able to get into FOF 2007 yet, I wonder if the AI is overcompensating when it's gameplanning against teams with weak run defenses (or strong pass defenses), and running more often then they should -- leading to less passes, leading to less production. Although, 5.3 yards per carry is pretty dang good, and should be enough to move you down the field.

[JeffW]

Although, 5.3 yards per carry is pretty dang good, and should be enough to move you down the field.

As I iterated above, not all 5.3 yards/carry averages are created equally. It is very possible that teams are gaining all their yards on a few select runs, but are getting stuffed on others and putting the offense into unfavorable down and distance situations that eventually lead to failed drives.

This is what happens in the Atlanta Falcons of the NFL's offense: Vick puts up huge yards/carry and Dunn is pretty good also, but they have a low success rate, so they're inconsistent at actually moving the ball downfield and sustaining drives.

[Leonidas]

In my own testing I really don't concern myself with wins and losses or how a team does against the rest of the league. I test a gameplan by improving its performance against the same team using AI gameplanning only. And in my testing I found the only consistent way I could improve overall performance was to attack the run. I found there was very little I could do in the gameplan to influence pass efficiency, but I could easily degrade run efficiency, hence it was the only means I had to improve performance.

That said, in my offensive testing, I have most certainly seen the phenomenon Jeff has observed happen for my own offensive gameplans. I found running outside more can definitely improve YPC, but I also found it had little or no impact to overall offensive efficiency. In fact, I have yet to find any offensive gameplan technique to improving the run offense that correlates to improving overall offensive efficiency. Not that running well is useless. Of course you want to get a good line and a good runner, but beyond that, I personally think you are wasting your time gameplanning anything special to it. Wanna improve the offense, then find the right passing gameplan for your QB and receivers. Wanna improve the defense, get some great coverage guys and gameplan to shut down the run.

[JeffW]

In my own testing I really don't concern myself with wins and losses or how a team does against the rest of the league.

The problem is, how do you measure efficiency outside of W-L record(maybe points scored/allowed converted to Pythagorean W-L is better to lower the variance). If you just measure the yardage numbers your data gets confounded by problems like the ones I talk about in OP.

[Leonidas]

The problem is, how do you measure efficiency outside of W-L record(maybe points scored/allowed converted to Pythagorean W-L is better to lower the variance). If you just measure the yardage numbers your data gets confounded by problems like the ones I talk about in OP.With my testing, I can test the same gameplan concepts with the worst team in the league or the best without a concern. I look at wins, but in comparison to what the team does with the AI. I could care less how a team ranks against the league, I am competing against the AI, or even a previous gameplan to make the most efficient gameplan I can devise. I think winning the league has more to do with talent, so I ignore that as an entirely seperate issue. My goal is to make the talent available operate its most efficiently.

I take a team, turn off injuries, run the same season with the same team over and over again 20 times. I start off with AI gameplanning on as my baseline. From there I try different gameplan ideas and test them over the same 20-season test. If the team scores more points and wins more games than with AI gameplanning, the gameplan is successful. All the other data is collected to evaluate trends and look for variables that can be controlled to make overall efficiency better. I have found strong trends and correlations this way.

The key seems to be improving yards per attempt. In every single test run I have ever made teams always win and score more when YPA improves. Yards per carry mean nothing to winning and scoring. Completion % is a secondary correlation to winning and scoring. About 90% of the time a team wins more when it's completion % is higher. I suspect that is really a component of the YPA phenomenon.

[JeffW]

With my testing...

We use the exact same testing strategy. I didn't specify, but I test wins compared to a control gameplan.

The key seems to be improving yards per attempt. In every single test run I have ever made teams always win and score more when YPA improves.

Are you sure that YPA doesn't improve when teams win and score more? ;)

We have to be careful not to confuse causation and correlation.

[Leonidas]

We use the exact same testing strategy. I didn't specify, but I test wins compared to a control gameplan.



Are you sure that YPA doesn't improve when teams win and score more? ;)

We have to be careful not to confuse causation and correlation.I broke out the stats by checking the numbers every season the team won more games than average, and the numbers every season a team scored more points than average and compared with the numbers when they were winning and scoring less. Usually completion percentages were higher when teams were winning and scoring more, but not always. That runs 80-90 % of the time. However, every single 20 season test block I have run (and I estimate I have done well over 30 of them) teams always have a higher YPA when they win more than average and when they score more than average. No other statistic stood out like that. It's almost eerie how true this holds. I keep expecting that one sim to defy.

So I turned this around when I started modelling defensive gameplans and the exact same thing happened. The team always wins more and allows fewer points when YPA is down. The problem there is I have yet to find a defensive gameplan that effectively limits YPA to take advantage of this. I have managed to find several offensive variants to improve YPA, but nothing on defense to do it consistently. I have found run defenses to lower yards per rush, but that doesn't necessarily equate to a better defense, just a better run defense. So I still think the only way you can attack on pass defense is get two cover guys who are maxed out on your coverage philosophy, not much you can do otherwise on the gameplan other than match the scheme with your guys' talents. I personally like BnR.

[JeffW]

Leonidas,

My point about causation/correlation is thus:

You're assuming that raising Y/A raises points scored and wins, but what if it's the other way around? At this point we haven't determined which causes which--just that they're highly correlated.

Winning teams are passing more in balanced situations--i.e. when the score is close and running and passing are both options and in run heavy situations(where the defense is playing Aggressive Run defense). Thus, their average attempt is going to gain more yards just by virtue of having more favorable opportunities compared to a worse team that is more often playing from behind and thus passing in situations where the defense is playing Aggressive Pass defense.

For the record, I think improving Y/A on both sides of the ball will very obviously improve points scored/allowed as well as win%, I am just cautioning you not to read too much into the results, because a bias similar to the one I mentioned in the original post is occuring here as well.

[Leonidas]

Leonidas,

My point about causation/correlation is thus:

You're assuming that raising Y/A raises points scored and wins, but what if it's the other way around? At this point we haven't determined which causes which--just that they're highly correlated.

Winning teams are passing more in balanced situations--i.e. when the score is close and running and passing are both options and in run heavy situations(where the defense is playing Aggressive Run defense). Thus, their average attempt is going to gain more yards just by virtue of having more favorable opportunities compared to a worse team that is more often playing from behind and thus passing in situations where the defense is playing Aggressive Pass defense.

For the record, I think improving Y/A on both sides of the ball will very obviously improve points scored/allowed as well as win%, I am just cautioning you not to read too much into the results, because a bias similar to the one I mentioned in the original post is occuring here as well.I think I am reading it perfectly. I have used this data to make a gameplan that sacrificed a bit of completion % to improve distance and YPA and have made it significantly improve offensive performance. In fact, every offense I have tried this with performed better. I even took it down and tested it on 2K4 and it worked. If it's just a coincidence, then it just coincidentally helped me find a gameplan that I really like and has beaten the pants off the AI gameplan in every 20 season block I have tested. And if I am being stupid for believing in the correlation, then my ignorance thus far has been quite blissful. :)

[Vinatieri for Prez]

I think you're still missing the point about correlations, rather than causes. First of all, this was discussed in a prior thread not to long ago; where I mentioned that not only in FOF, but also in the NFL, YPA number has been shown to be one of the highest statistical indicators amongst successful teams. However, you can't go out and maximize YPA with a YPA gameplan. YPA is no different than points scored/against differential. Obviously, the better differential, the more successful the team. But it's not like you can go out and say "ok, which gameplan will get me the biggest point differential." Rather, you go out and get good players, pick plans that fit their talents, and lo and behold you will see that you will have better point differential and YPA (i.e. YPA or point differential is a very high correlation with winning) My point being is that by going with a heavy run offense (with the right talent) it just as very well get you a better point differential and YPA, or vice versa. For instance, in the NFL, defenses will stack the box so that when you eventually decide to throw the ball, it likely will be completed for a longer gain.