When it comes to prediction, we are quick to forget how wrong we often are. So, in a post from earlier this year, I compiled early season NFL power rankings from multiple sources over multiple years and attempted to objectively measure how good each ranking system is at predicting wins (i.e. a more comprehensive version of the Panthers example above).
I intend to do the same comparison for 2014, so I figured I would go ahead and archive the week 4 rankings in this post, and then check back on them when the season has finished. Note that when I say "week 4 rankings", they are rankings compiled after week 4 of the season has been completed (and prior to Thursday night's Packers-Vikings blowout). Here are the ranking systems I'm comparing:
- SRS - What's known as the Simple Ranking System, which is based solely on margin of victory and strength of schedule.
- DVOA - Football Outsiders' DVOA rankings, a proprietary ranking system based on analysis of detailed play by play data.
- 538 - A new entrant, FiveThirtyEight's ELO rankings. The ELO ranking system was originally developed for chess, but has been extended to the NFL by Nate Silver.
- ESPN - ESPN's weekly NFL power rankings.
- Market - The betting market rankings found here at inpredictable.
- AFA - The Efficiency rankings from Advanced Football Analytics, which are based on per play stats (yards per attempt, success rates), with an adjustment for opponent strength.
Here is a table summarizing the rankings for each team.
The Patriots have the largest variance in ranking across the six different approaches. The three purely stat-based methods (SRS, DVOA, and AFA) have the Patriots the lowest, befitting their abysmal statistical performance over the first four weeks. 538, ESPN, and the Market have them somewhat higher. This makes sense as those three rankings factor in prior season results, and thus do not blind themselves to the Pats' recent historical dominance (see this amazing chart by FiveThirtyEight's Reuben Fischer-Baum in Benjamin Morris' week 4 NFL column).
As a reminder, we will use these rankings to measure their ability to predict future wins this season (excluding week 17 and the playoffs). Our standard for accuracy is Spearman's rank correlation coefficient, which scales from -100% to 100%, and is a measure of how closely two ordinal rankings agree with each other (a coefficient of 100% implies perfect agreement).
But that is several months away. In the meantime, we can use Spearman's coefficient to assess how closely the various rankings agree with each other. See the table below (a higher coefficient indicates a higher level of consistency between the two rankings).
As is often the case, the AFA rankings find themselves most at odds with the other ranking systems, with an average agreement of just 60%. The closest agreement appears to be between my market rankings and ESPN's, with a 90% Spearman coefficient.
In addition to the final season reckoning, I plan on checking in again after week 8 to see both how well each ranking system is doing, as well has how each set of rankings has shifted with the benefit of four more games.