The Upset, Part II: All over the place

After discussing why we rank, it's time to say just how good each scheme is.  For the purposes of this study, I've looked at the rankings as of 1 Jan 2013.

Retrodiction
or, how often did the team that won end up ranked lower?

This is the bit that causes the most confusion.  "But we beat them, why aren't we higher?"  Well, ranking algorithms generally aren't actually written to minimize this.  They're written to be more concerned about other things, and hope this comes along for the ride.  A scheme has been written, called Min-V, which is primarily concerned about minimizing retrodiction error.  More on its algorithm later.

How do Derby Chart, the European Roller Derby Rankings, and Min-V stack up in this category, as well as predictive ability?

	DC	ERDR	Min-V	bouts
Past upsets	70	55	9	272
Past upset %	25.7%	20.2%	3.3%	272
Predict upsets	16	15	17	45
Predict upset %	35.6%	33.3%	37.8%	45

Turns out, both rankings do a poor job of retrodicting bouts.  Only the Min-V system, with the sole purpose of minimizing retrodiction errors, has a low upset percentage.  For prediction, all three give at best a 2-in-3 chance of being correct.

Conclusion

Don't trust the rankings too much, unless they're Min-V.  And Min-V looks like this:

1	London Rollergirls	20.911
2	Gent GO-GO Roller Girls [B]	20.713
3	London Rollergirls [B]	20.711
4	Paris Roller Girls	20.703
5	Bear City Roller Derby	20.697
6	Rainy City Roller Girls	20.672
7	Stockholm Roller Derby	20.668
8	Hellfire Harlots	20.612
9	Middlesbrough Milk Rollers	20.602
10	Glasgow Roller Derby [B]	20.6
11	Brighton Rockers	20.592
12	Glasgow Roller Derby	20.587
13	London Rollergirls [C]	20.58
14	Helsinki Roller Derby	20.572
15	Gent GO-GO Roller Girls	20.568
16	Leeds Roller Dolls	20.558
16	Tiger Bay Brawlers	20.558
18	Auld Reekie Roller Girls	20.548
18	Crime City Rollers	20.548
20	Central City Rollergirls	20.537
21	Dublin Roller Girls	20.502
21	Hot Wheel Roller Derby	20.502
23	Newcastle Roller Girls [B]	20.5
23	Leeds Roller Dolls [B]	20.5
23	Manchester Roller Derby	20.5
26	Bear City Roller Derby [B]	20.498
27	Copenhagen Roller Derby	20.432
28	Kallio Rolling Rainbow	20.427
29	Sheffield Steel Roller Girls [B]	20.4
30	Royal Windsor Rollergirls	20.232
31	Southern Discomfort	20.2
32	Lincolnshire Rolling Thunder	20.1
32	Quad Guards	20.1
32	Ruhrpott Roller Girls	20.1
35	Tyne & Fear	20
36	Expendables	19.9
37	Roller Derby Bordeaux Club	10.6
38	Roller Girls of the Apocalypse	10.5
38	Herault Derby Girlz	10.5
40	Cork City Firebirds	10.488
41	Bristol Roller Derby	10.442
42	Birmingham Blitz Dames	10.437
43	London Rockin Rollers [B]	10.421
44	Stuttgart Valley Rollergirlz [B]	10.402
45	Paris Roller Girls [B]	10.4
45	Roller Derby Rennes	10.4
47	One Love Roller Dolls	10.398
48	Lutèce Destroyeuses - Paris	10.3
49	Brussels Derby Pixies	10.2
49	Roller Derby Metz Club	10.2
51	MRD: New Wheeled Order	10.1
52	Newcastle Roller Girls	1
53	Crime City Rollers [B]	0.988
54	Liverpool Roller Birds	0.9
54	Seaside Sirens Roller Girls	0.9
56	Sheffield Steel Roller Girls	0.888
56	London Rockin Rollers	0.888
56	Dolly Rockit Rollers	0.888
59	Stuttgart Valley Rollergirlz	0.878
59	Romsey Town Rollerbillies	0.878
61	Croydon Roller Derby	0.8
61	Rainy City Roller Girls [B]	0.8
63	Big Bucks High Rollers	0.798
64	Lincolnshire Bombers	0.788
65	Tiger Bay Brawlers [B]	0.7
66	Vienna Roller Girls	0.6
67	Rockcity Rollers	0.5
67	Milton Keynes Concrete Cows	0.5
67	Barockcity Rollerderby	0.5
67	Amsterdam Derby Dames	0.5
71	Bristol Roller Derby [B]	0.4
71	Rebellion Roller Derby	0.4
71	Dublin Roller Girls [B]	0.4
71	Harbor Girls	0.4
71	Portsmouth Roller Wenches	0.4
71	Liverpool Roller Birds [B]	0.4
71	Rotterdam Death Row Honeys	0.4
78	Munich Rolling Rebels	0.3
78	Blackland Rockin'K-Rollers	0.3
78	Bembel Town Roller Girls	0.3
78	Oxford Roller Derby	0.3
78	Royal Windsor Rollergirls [B]	0.3
78	Birmingham Blitz Dames [B]	0.3
78	Bad Bunny Rollers	0.3
85	Inhuman League	0.2
85	South West Angels of Terror	0.2
85	Manchester Roller Derby [B]	0.2
85	Bedfordshire Roller Girls	0.2
85	Mean Valley Roller Girls	0.2
85	Copenhagen Roller Derby [B]	0.2
85	Belfast Roller Derby	0.2
85	Hereford Roller Girls	0.2
85	Roller Derby Karlsruhe	0.2
85	Roller Derby Lyon	0.2
85	Dirty River Roller Grrrls	0.2
85	One Love Roller Dolls [B]	0.2
85	Gothenburg Roller Derby	0.2
85	Namur Roller Girls	0.2
99	Swansea City Roller Derby	0.1
99	Roller Derby Belfort	0.1
99	Crash Test Brummies	0.1
99	Evolution Rollergirls	0.1
99	Kent Roller Girls	0.1
99	Frankfurt Roller Derby	0.1
99	Dundee Roller Girls	0.1
99	Plymouth City Roller Girls	0.1
99	Dorset Roller Girls	0.1
99	Roller Derby Toulouse [B]	0.1
99	Norfolk Brawds	0.1
99	Helsinki Roller Derby [B]	0.1
99	Bruising Banditas	0.1
99	Kallio Rolling Rainbow [B]	0.1
99	Nantes Derby Girls	0.1
99	Roller Derby Metz Club [B]	0.1
99	Lincolnshire Bombers [B]	0.1
99	Dom City Dolls	0.1
99	Fierce Valley Roller Girls	0.1
99	Tampere Roller Derby	0.1
99	Central City Rollergirls [C]	0.1
99	Roller Derby Calaisis	0.1
99	Furness Firecrackers	0.1
99	Roller Derby Grenoble	0.1
99	Hell's Belles	0.1
99	Nidaros Roller Derby	0.1
99	Porto Roller Derby	0.1
126	Seaside Sirens Roller Girls [B]	0
126	Severn Roller Torrent	0
126	Shoetown Slayers	0
126	Barcelona Roller Derby	0
126	Les Quads de Paris	0
126	Imposters Roller Girls	0
126	Roller Derby Arras	0
126	Hell's Ass Derbygirls	0
126	Fair City Rollers	0
126	Brighton Rockers [B]	0
126	Velvet Sluts	0
126	Wolverhampton Honour Rollers	0
126	Nottingham Roller Girls	0
126	Kernow Rollers	0
126	Wirral Whipiteres	0
126	Wakey Wheeled Cats	0
126	Wiltshire Roller Derby	0
126	Vendetta Vixens	0
126	Roller Derby Angoulême	0
126	Eastside RocknRollers	0
126	Tenerife Roller Derby	0
126	Aarhus Derby Dames	0
126	Dolly Rockit Rollers [B]	0
126	Cardiff Roller Collective	0
126	Cornwall Roller Derby	0
126	Cherry Blood	0
126	Marseille Roller Derby Club	0
126	Spiders Black Widows	0
126	Oslo Roller Derby	0
126	Central City Rollergirls [B]	0
126	Roller Girls of the Apocalypse [B]	0
126	Amsterdam Derby Dames [B]	0
126	Auld Reekie Roller Girls [B]	0
126	Jakey Bites	0
126	Lahti Roller Derby	0
126	Royal Swedish Roller Derby	0
126	Luleå Roller Derby	0
126	Preston Roller Girls	0
126	Graveyard Queens Cologne	0
126	Lahti Roller Derby [B]	0
126	Roller Derby Lorient	0
126	Roller Derby Lille	0
126	Porvoo	0
126	Dresden Pioneers	0
126	Bairn City Rollers	0
126	Voodoo Vixens Besançon	0
126	Prague City Roller Derby	0
126	Plymouth City Roller Girls [B]	0
126	Grin n Barum	0
126	Kouvola Rock n Rollers	0
126	Southern Discomfort [B]	0
126	The Switchblade RollerGrrrls	0
126	Tester	0
126	Stockholm Roller Derby [B]	0
126	Dock City Rollers	0
126	Nantes Derby Girls [B]	0
126	Valencia Roller Derby	0
126	South Wales Silures	0
126	Panam Squad	0
126	Red Lion Roller Derby	0
126	Nought	0
126	Zurich City Rollergirls	0
126	Montpellier Derby Club	0
126	Limerick Roller Derby	0
126	Hulls Angels Roller Dames	0
126	Middlesbrough Milk Rollers [B]	0
126	B.M.O Roller Derby Girls	0
126	Roller Derby Toulouse	0
126	Big Bucks High Rollers [B]	0
126	Tyne & Fear [B]	0
126	Surrey Roller Girls	0
126	Kamiquadz	0
126	Milton Keynes Quads of War	0
126	Roller Derby Avingon	0
126	Bourne Bombshells	0
126	Kent Roller Girls [B]	0
126	Granite City Roller Girls	0