You can't take a small sample of something, find that there is a good distribution in the sample, and infer that you have precision in the answer. Sometimes you get lucky.... very lucky.
That's exactly it! How often will you get lucky in Vegas? Statistically, not very often, because that's how Vegas is designed. These tests are similar, but we could get lucky. That being said, given the odds of getting lucky compared to the test being correct, I would go with the test being right because the size of samples compared to the increase and SD indicate it's very likely to be correct. Maybe not... But I never win in Vegas.
Originally Posted by FormulaTwo
I think if i could get that type of FE i would have no problem driving a dildo shaped car.
The difference between standard deviation and improvement will dictate how many runs ya need to do IIRC. Where's Bill when ya need 'em? Well, I think this is right, and in the case of the Camry, assuming a SD of ~.1mpg (I think it's actually less) and a change of .4mpg, 2 runs for each gives an 80 chance that an 0.05 level test of significance will find a statistically significant difference between two sample means. I think...
The nice thing about a SG is the amount of resolution it affords. W/o that, we would have a much higher SD and need to do more runs to get a statistically significant result.
Yes, you are right. The question you are asking is "Is there a difference?", and that calculator tells you how many runs you need. It assumes that your standard deviation before and after the mod will be the same. E-mail me if any other questions come up. I'm not around much anymore. In an old thread with a lot of stats stuff, I posted a better calculator where you could put in your data and it would tell you if the two samples were different.