What's wrong with this picture?

Post Reply
Velocomp
Velocomp CEO
Posts: 7793
Joined: Fri Jan 11, 2008 8:43 am

What's wrong with this picture?

Post by Velocomp »

When you're a new guy on the block, you want to prove yourself. So we were fascinated when we stumbled across this data, taken from the yet-to-be-released Limits power meter, showing their performance against an SRM.

This is the first data they have presented since announcing their project 8 months ago. They have taken a lot of flak from DC Rainmaker (properly in our view) for making bold claims without any supporting data.

Their graph title (proudly?) says "LIMITS vs SRM Power 0.94 correlation coefficient". We'll assume their statisticians are correct.

But we're cyclists, not statistics majors, so we looked at the data from a cyclist's point of view...

The ride shown is about 9 ½ minutes long...in our view not long enough to prove anything.

For the first "long" interval, seconds 51 to 101, the Limits reads 20% low (200 vs 250W). NOTE: all of this analysis is by eyeball, so exact results may differ slightly.

The first surge, at second 115, Limits is 29% low (250 vs 350).

The second long interval, 126 to 226 seconds, the Limits is about 19% low, about 40W less than the SRM.

The second surge, at 276 seconds, the Limits is 37% low (300 vs 480)

The third interval, 276 to 326 seconds, Limits is consistently low, and completely misses the SRM surge at second 320.

Starting at second 360, Limits again misses the surge by 23% low (290 vs 375)

Good news: when the cyclist backs off on power, between 376 and 426 the two devices are close.

But in the final seconds of the short ride (426 to 576) , where there are six surges of power, Limits is 58% low on one burst at second 476, and 56% low at second 501.

We honestly don't know many power meter users who would accept this quality of performance, and even fewer who could figure out how to benefit from it.

Based on our cycling-oriented view of power and the data presented here's a title for their graph that we'd suggest: "LIMITS vs SRM Power: consistently low performance" :-)
Attachments
Limits vs SRM
Limits vs SRM
Screen Shot 2016-01-14 at 9.53.10 AM.png (175.3 KiB) Viewed 4279 times
John Hamann
EHB
Posts: 129
Joined: Sun Oct 26, 2014 5:52 pm

Re: What's wrong with this picture?

Post by EHB »

It has been a while since I've dabbled in statistics, but there are several types of correlation coefficient. I'm guessing they are reporting a Pearson product moment correlation coefficient.

Firstly that is absolutely the wrong sort of graph to show a correlation. If they have involved a statistician I suspect they are pulling their hair out.

Secondly, it is the wrong statistical test anyway. In this case we are to assume we have a gold standard (the SRM) and that it is correct. It may not be true, but it is the best (gold) we have.

Thirdly, 0.94 isn't great anyway.

I'd do a Bland-Altman plot on this, assuming the SRM is correct, with the correct sort of graph. That would give us an average error, how confident we are about the average-error and what ranged of error will likely happen 95% of the time (assuming error is normally distributed).
Pete
Posts: 110
Joined: Wed Mar 05, 2014 8:09 am
Location: Melburn, AUS

Re: What's wrong with this picture?

Post by Pete »

What's wrong with the picture is that it's missing iBike data as comparison (or any other point of reference)
Velocomp
Velocomp CEO
Posts: 7793
Joined: Fri Jan 11, 2008 8:43 am

Re: What's wrong with this picture?

Post by Velocomp »

Pete wrote:What's wrong with the picture is that it's missing iBike data as comparison (or any other point of reference)
SRM data is in the graph.
John Hamann
Pete
Posts: 110
Joined: Wed Mar 05, 2014 8:09 am
Location: Melburn, AUS

Re: What's wrong with this picture?

Post by Pete »

Velocomp wrote:
Pete wrote:What's wrong with the picture is that it's missing iBike data as comparison (or any other point of reference)
SRM data is in the graph.
Yes I meant they should have other units to show how good their thing is compared to them
Like this:

Image
Post Reply