I was doing some research last night, and read a number of academic papers on the ranking of sports teams where there is no balanced schedule. (Balanced schedule meaning that each team plays each other team, like in the Premiership.)
The thing that the articles made me realize is that there is no such thing as a "perfect" ranking scheme. What would "perfect" mean? No upsets, maybe? Well, there will always be upsets in sport, so that really won't work.
Every ranking scheme, it turns out, has some subjective basis. The opinions of the author of the scheme influence how various elements are included. As a simple example consider this:
It's easier to consider points as a percentage, rather than a ± number. Thus, one constructs a points percentage.
- Considers not only how well XRD does, but how difficult the teams XRD plays are. This should be obvious, and most derby ranking schemes feature it in one sense or another. It just never hurts to mention it again.
- Allows bouts to expire. Most other sports have seasons, and teams receive a clean slate at the beginning of a new season. Even the other major seasonless sport, Cricket, has rankings with an expiry date. After 36 months, a match is no longer used in the calculation of the test match rankings.
- Is an open formula. Even though it may require an advanced maths degree to calculate, at least a fair ranking scheme can explain itself. This is not true of all schemes, go see the American college football BCS poll to see a closed formula.
- Can be updated on a regular basis. The WFTDA rankings are based on a subjective poll. Whilst this is a good way of dealing with the awkward schedules of roller derby leagues, it does limit the number of times a ranking can be measured due to the logistics of the poll. A fair scheme would allow a team to wait only a few days to see how that last win, or heartbreaking loss, affected their standings in the table.
- Continually compound data. Thus if XRD defeat a highly ranked team that subsequently falls far, that win should count as less of an upset than it did initially.
- This may be exceedingly difficult to do, but it would certainly be nice.