Friday, October 18, 2013

Early Season Power Rankings - Follow Up

This is a follow up to last week's post on early season power rankings. In that post, I attempted to measure the accuracy of various NFL power rankings by comparing the team rankings after week 4 of the season to each team's remaining wins for weeks 5-16.

The original dataset was for seasons 2009-2012, and just looked at the week 4 rankings. I have now added seasons 2007 and 2008, plus a look at how the rankings after week 8 correlated with future wins. As a reminder, higher percentages are better, with 100% meaning perfect correlation between the ranking and future wins (the metric used is the Spearman rank correlation coefficient).

Week 5 Ranking Correlation to Future Wins
ranking average 2007 2008 2009 2010 2011 2012
espn 47% 55% 42% 51% 55% 43% 39%
dvoa 50% 57% 45% 47% 46% 41% 65%
ans 40% 50% 42% 51% 15% 49% 32%
market 54% 68% 36% 67% 45% 52% 56%
srs 52% 70% 46% 47% 38% 56% 54%

Week 8 Ranking Correlation to Future Wins
ranking average 2007 2008 2009 2010 2011 2012
espn 46% 58% 42% 51% 43% 41% 41%
dvoa 53% 77% 53% 59% 44% 35% 46%
ans 49% 70% 47% 57% 38% 39% 42%
market 55% 62% 50% 62% 56% 55% 42%
srs 53% 75% 51% 55% 46% 42% 50%


Keep it Simple

The performance of the Simple Ranking System (SRS) continues to impress, either matching or outperforming the more sophisticated DVOA and ANS models. The market ranking still appears to be the most accurate.

More data makes you dumber

Interesting to note that while all of the quantitative rankings improved going from week 4 to week 8 (particularly the Advanced NFL Stats efficiency model), the qualitative ESPN ranking took a small step backward. It's hard to read too much into such a small difference, but it is consistent with Ken Pomeroy's observations on the AP rankings for college basketball. For the 1990-2010 academic years, he compared tournament performance of the pre-season AP #1 to the AP #1 immediately prior to the tournament. In general the pre-season #1 fared better in the tournament (out of 21 seasons, the pre-season #1 won it all 6 times, compared to just 3 for the pre-tournament #1).

It's counter-intuitive, but has a simple explanation. As you get further into the season, most poll-based rankings tend to become retrodictive, rather than predictive. In other words, they "reward" teams for past wins, rather than judge them on their ability to create future wins. Bill Parcells statements not withstanding, a team's win-loss record is not always the best predictor of future success.

1 comment:

  1. Great stuff. Have you tried regressing the different models together to see what combination of them does the best? It should be the case that a meta-ranking which is composed of, say, 70% market + 20% srs + 10% dvoa outperforms any of the rankings individually.

    ReplyDelete