Saturday, October 4, 2014

NFL Week 4 Power Ranking Roundup

Cam Newton 2014Last year the Carolina Panthers, after starting the season 1-2, led the NFL in win percentage from weeks five through sixteen, winning 10 of their next 12. Who saw that coming? Prior to their 10-2 run, ESPN had the Panthers at #21 in their weekly power rankings. My own betting market rankings had them ranked 18th. Football Outsiders' DVOA was closer to the mark, ranking the Panthers 6th. What's known as the Simple Ranking System was even closer though, ranking the Panthers 3rd.

When it comes to prediction, we are quick to forget how wrong we often are. So, in a post from earlier this year, I compiled early season NFL power rankings from multiple sources over multiple years and attempted to objectively measure how good each ranking system is at predicting wins (i.e. a more comprehensive version of the Panthers example above).

I intend to do the same comparison for 2014, so I figured I would go ahead and archive the week 4 rankings in this post, and then check back on them when the season has finished. Note that when I say "week 4 rankings", they are rankings compiled after week 4 of the season has been completed (and prior to Thursday night's Packers-Vikings blowout). Here are the ranking systems I'm comparing:
  • SRS - What's known as the Simple Ranking System, which is based solely on margin of victory and strength of schedule.
  • DVOA - Football Outsiders' DVOA rankings, a proprietary ranking system based on analysis of detailed play by play data.
  • 538 - A new entrant, FiveThirtyEight's ELO rankings. The ELO ranking system was originally developed for chess, but has been extended to the NFL by Nate Silver.
  • ESPN - ESPN's weekly NFL power rankings.
  • Market - The betting market rankings found here at inpredictable.
  • AFA - The Efficiency rankings from Advanced Football Analytics, which are based on per play stats (yards per attempt, success rates), with an adjustment for opponent strength.
Here is a table summarizing the rankings for each team.

team SRS DVOA 538 ESPN MARKET AFA
ARZ 2 8 6 4 10 10
ATL 12 4 22 17 16 22
BAL 5 5 8 7 9 9
BUF 13 10 24 23 22 24
CAR 27 27 14 20 21 25
CHI 18 14 19 15 19 20
CIN 1 1 5 2 3 1
CLE 19 16 29 25 28 16
DAL 14 13 9 10 11 18
DEN 6 3 2 3 2 4
DET 11 7 17 8 7 3
GB 15 6 16 12 4 5
HOU 17 24 25 13 20 21
IND 7 17 10 11 5 17
JAC 32 32 32 31 32 31
KC 9 18 12 14 17 14
MIA 24 20 21 22 23 6
MIN 22 26 20 24 26 26
NE 26 23 7 16 14 27
NO 20 21 15 21 8 7
NYG 8 11 18 19 15 11
NYJ 28 25 27 26 25 12
OAK 30 30 31 32 31 29
PHI 16 9 11 6 13 28
PIT 23 15 13 18 18 23
SD 4 12 4 5 12 8
SEA 3 2 1 1 1 2
SF 10 19 3 9 6 19
STL 29 29 23 30 27 30
TB 31 31 28 29 30 32
TEN 21 28 26 28 29 15
WAS 25 22 30 27 24 13

The Patriots have the largest variance in ranking across the six different approaches. The three purely stat-based methods (SRS, DVOA, and AFA) have the Patriots the lowest, befitting their abysmal statistical performance over the first four weeks. 538, ESPN, and the Market have them somewhat higher. This makes sense as those three rankings factor in prior season results, and thus do not blind themselves to the Pats' recent historical dominance (see this amazing chart by FiveThirtyEight's Reuben Fischer-Baum in Benjamin Morris' week 4 NFL column).

As a reminder, we will use these rankings to measure their ability to predict future wins this season (excluding week 17 and the playoffs). Our standard for accuracy is Spearman's rank correlation coefficient, which scales from -100% to 100%, and is a measure of how closely two ordinal rankings agree with each other (a coefficient of 100% implies perfect agreement).

But that is several months away. In the meantime, we can use Spearman's coefficient to assess how closely the various rankings agree with each other. See the table below (a higher coefficient indicates a higher level of consistency between the two rankings).

SRS DVOA 538 ESPN MARKET AFA Avg
SRS 84% 73% 86% 80% 66% 78%
DVOA 84% 63% 81% 79% 65% 74%
538 73% 63% 88% 86% 46% 71%
ESPN 86% 81% 88% 90% 58% 81%
MARKET 80% 79% 86% 90% 66% 80%
AFA 66% 65% 46% 58% 66% 60%

As is often the case, the AFA rankings find themselves most at odds with the other ranking systems, with an average agreement of just 60%. The closest agreement appears to be between my market rankings and ESPN's, with a 90% Spearman coefficient.

In addition to the final season reckoning, I plan on checking in again after week 8 to see both how well each ranking system is doing, as well has how each set of rankings has shifted with the benefit of four more games.

4 comments:

  1. Is the table of correlations based on just this season, or the past few, as your previous articles on this topic have covered?

    ReplyDelete
  2. Did you ever run an update on this after Week 8? Couldn't find it on your site. Thanks!

    ReplyDelete
    Replies
    1. I was not able to get to it, unfortunately. At this point, I think I'll just wait until week 16 is complete.

      Delete