The Nature of Rankings

I was doing some research last night, and read a number of academic papers on the ranking of sports teams where there is no balanced schedule.  (Balanced schedule meaning that each team plays each other team, like in the Premiership.)

The thing that the articles made me realize is that there is no such thing as a "perfect" ranking scheme.  What would "perfect" mean?  No upsets, maybe?  Well, there will always be upsets in sport, so that really won't work.




Every ranking scheme, it turns out, has some subjective basis.  The opinions of the author of the scheme influence how various elements are included.  As a simple example consider this:


It's easier to consider points as a percentage, rather than a ± number.  Thus, one constructs a points percentage.

This one's simple, it's just a percentage of points the team in question won.  However, in a whitewash, the losing team doesn't get credit for showing up, and low-scoring games tend to overemphasize the difference between the two teams.

Then there's this one.  Fixes the low-scoring game issues, and de-emphasizes blow out games.  I think that the 100th point contains less information about the two teams' relative abilities than the 10th point.  Thing is, what if you disagree?

Maybe you think that there's more information in the blow out then there is in a close game.  I disagree with you, but this is what I mean--rankings are subjective!

Besides, these three formulae only show how the outcome of a game is formulated into usable data, not how to do an overall ranking.  There is much subjectivity in how that is done, as well!

So, since we can't do truly objective, can we at least do fair?  Again, there's subjectivity in fair, but let me make a case here.  A fair ranking of team XRD:

1. Considers not only how well XRD does, but how difficult the teams XRD plays are.  This should be obvious, and most derby ranking schemes feature it in one sense or another.  It just never hurts to mention it again.
2. Allows bouts to expire.  Most other sports have seasons, and teams receive a clean slate at the beginning of a new season.  Even the other major seasonless sport, Cricket, has rankings with an expiry date.  After 36 months, a match is no longer used in the calculation of the test match rankings.
3. Is an open formula.  Even though it may require an advanced maths degree to calculate, at least a fair ranking scheme can explain itself.  This is not true of all schemes, go see the American college football BCS poll to see a closed formula.
4. Can be updated on a regular basis.  The WFTDA rankings are based on a subjective poll.  Whilst this is a good way of dealing with the awkward schedules of roller derby leagues, it does limit the number of times a ranking can be measured due to the logistics of the poll.  A fair scheme would allow a team to wait only a few days to see how that last win, or heartbreaking loss, affected their standings in the table.

A good system would also:

5. Continually compound data.  Thus if XRD defeat a highly ranked team that subsequently falls far, that win should count as less of an upset than it did initially.
• This may be exceedingly difficult to do, but it would certainly be nice.
6. Be based on simple data.  Sure, a system may produce valid results based on lead jams, total jams, total penalties, points, number of minors, etc., but such a scheme would be very difficult to implement.  A scheme based on points alone would allow quick entry, and far more transparent results.  It would also allow the calculation of rankings for bouts that occurred before the scheme was implemented.
7. Be expandable.  If a new league joins the ranking, it should not take any special effort to add it, and it should be fairly represented after a bout or two.  For me, I think 2 bouts is a fair minimum to be featured in a ranking.
8. Produce a low number of upsets.  Sure, upsets are part of the nature of sport, but if more than 50% of bouts are upsets, than the ranking scheme is wrong.  30% is probably a good number for derby, although the lower the better.

So, can a ranking scheme be both good and fair?  I hope so!  It would be nice to apply such a scheme to roller derby, even just on the scale of Europe or the UK & Ireland alone.  

Then we could do a Tournament of Champions, featuring the best league from each region.  Sure, there's already Track Queens, and I'm looking forward to that like a kid to Christmas.  It's going to be a blast!  But its invitations based on subject ratings.  Imagine the drama if there were a published table and leagues were told something along the lines of "top 2 of each region on Sept 1 go!"  Then all those August bouts for teams in positions 1-5 become massively interesting.  Who's going to Berlin?  Who's going to have their hearts broken?  Will we see [insert favourite rivalry] in the tournament?

Wow, how exciting that would be.  And having a trusted good and fair ranking scheme would make it possible.  So is such a scheme a possibility?

And, on that note,

Roll on!