NFL Week 4 Power Ranking Roundup

Last year the Carolina Panthers, after starting the season 1-2, led the NFL in win percentage from weeks five through sixteen, winning 10 of their next 12. Who saw that coming? Prior to their 10-2 run, ESPN had the Panthers at #21 in their weekly power rankings. My own betting market rankings had them ranked 18th. Football Outsiders' DVOA was closer to the mark, ranking the Panthers 6th. What's known as the Simple Ranking System was even closer though, ranking the Panthers 3rd.

When it comes to prediction, we are quick to forget how wrong we often are. So, in a post from earlier this year, I compiled early season NFL power rankings from multiple sources over multiple years and attempted to objectively measure how good each ranking system is at predicting wins (i.e. a more comprehensive version of the Panthers example above).

I intend to do the same comparison for 2014, so I figured I would go ahead and archive the week 4 rankings in this post, and then check back on them when the season has finished. Note that when I say "week 4 rankings", they are rankings compiled after week 4 of the season has been completed (and prior to Thursday night's Packers-Vikings blowout). Here are the ranking systems I'm comparing:

• SRS - What's known as the Simple Ranking System, which is based solely on margin of victory and strength of schedule.
• DVOA - Football Outsiders' DVOA rankings, a proprietary ranking system based on analysis of detailed play by play data.
• 538 - A new entrant, FiveThirtyEight's ELO rankings. The ELO ranking system was originally developed for chess, but has been extended to the NFL by Nate Silver.
• ESPN - ESPN's weekly NFL power rankings.
• Market - The betting market rankings found here at inpredictable.
• AFA - The Efficiency rankings from Advanced Football Analytics, which are based on per play stats (yards per attempt, success rates), with an adjustment for opponent strength.

Here is a table summarizing the rankings for each team.

team	SRS	DVOA	538	ESPN	MARKET	AFA
ARZ	2	8	6	4	10	10
ATL	12	4	22	17	16	22
BAL	5	5	8	7	9	9
BUF	13	10	24	23	22	24
CAR	27	27	14	20	21	25
CHI	18	14	19	15	19	20
CIN	1	1	5	2	3	1
CLE	19	16	29	25	28	16
DAL	14	13	9	10	11	18
DEN	6	3	2	3	2	4
DET	11	7	17	8	7	3
GB	15	6	16	12	4	5
HOU	17	24	25	13	20	21
IND	7	17	10	11	5	17
JAC	32	32	32	31	32	31
KC	9	18	12	14	17	14
MIA	24	20	21	22	23	6
MIN	22	26	20	24	26	26
NE	26	23	7	16	14	27
NO	20	21	15	21	8	7
NYG	8	11	18	19	15	11
NYJ	28	25	27	26	25	12
OAK	30	30	31	32	31	29
PHI	16	9	11	6	13	28
PIT	23	15	13	18	18	23
SD	4	12	4	5	12	8
SEA	3	2	1	1	1	2
SF	10	19	3	9	6	19
STL	29	29	23	30	27	30
TB	31	31	28	29	30	32
TEN	21	28	26	28	29	15
WAS	25	22	30	27	24	13

The Patriots have the largest variance in ranking across the six different approaches. The three purely stat-based methods (SRS, DVOA, and AFA) have the Patriots the lowest, befitting their abysmal statistical performance over the first four weeks. 538, ESPN, and the Market have them somewhat higher. This makes sense as those three rankings factor in prior season results, and thus do not blind themselves to the Pats' recent historical dominance (see this amazing chart by FiveThirtyEight's Reuben Fischer-Baum in Benjamin Morris' week 4 NFL column).

As a reminder, we will use these rankings to measure their ability to predict future wins this season (excluding week 17 and the playoffs). Our standard for accuracy is Spearman's rank correlation coefficient, which scales from -100% to 100%, and is a measure of how closely two ordinal rankings agree with each other (a coefficient of 100% implies perfect agreement).

But that is several months away. In the meantime, we can use Spearman's coefficient to assess how closely the various rankings agree with each other. See the table below (a higher coefficient indicates a higher level of consistency between the two rankings).

	SRS	DVOA	538	ESPN	MARKET	AFA	Avg
SRS		84%	73%	86%	80%	66%	78%
DVOA	84%		63%	81%	79%	65%	74%
538	73%	63%		88%	86%	46%	71%
ESPN	86%	81%	88%		90%	58%	81%
MARKET	80%	79%	86%	90%		66%	80%
AFA	66%	65%	46%	58%	66%		60%

As is often the case, the AFA rankings find themselves most at odds with the other ranking systems, with an average agreement of just 60%. The closest agreement appears to be between my market rankings and ESPN's, with a 90% Spearman coefficient.

In addition to the final season reckoning, I plan on checking in again after week 8 to see both how well each ranking system is doing, as well has how each set of rankings has shifted with the benefit of four more games.