Serious testing takes time and energy.

[95metro]

A-B-A testing is the ONLY method of properly quantifying the possible gains or losses of a "fuel-saving" (or power producing) device.

The absolute best method is usually not available to us. This would consist of a dyno, a rolling road, and a test lab with absolute controlled environment. This would remove most of the variables that on-road tests are generally prone to.

On-road tests with some form of data collection (ie - ScanGauge, SuperMID, data acquisition computer/software) are always at the mercy of the elements. Temperature, humidity, wind, and (most of all) the driver are variables that cannot be determined as to their affects on a given test.

However, a calm, warm day and a relatively flat road (or predetermined "track") will provide some data to at least make educated assumptions on. At the very least it puts theories in the "plausible" category.

For sake of reference I will refer to the chosen roadway as a "track".

"A" will refer to the test vehicle in "stock" or "current" (if already modified) condition.

"B" will refer to the test vehicle with the "device" or "additive" installed.

Weather and track conditions must be recorded before each run. For optimum results the test vehicle should be warmed to operating temperture and then driven on the track numerous times in its "A" condition to establish a baseline and/or margin of error between runs. I personally feel that multiple runs are necessary since environment and driving style could cause fluctuations in fuel economy during each run.

Example: Test vehicle in "A" condition
Run1: 42.5 mpg
Run2: 43.3 mpg
Run3: 42.1 mpg
Run4: 42.9 mpg

This establishes some form of baseline for what the test vehicle is capable of on this particular day at this particular time. The margin of error is 1.2 mpg between the best and worst runs. Now, let's attach the device/additive in question and do more runs (it may even be best to drive the test vehicle normally for a set amount of time before performing each set of runs).

Example: Test vehicle in "B" condition
Run1: 43.5 mpg
Run2: 42.8 mpg
Run3: 43.1 mpg
Run4: 42.5 mpg

In this particular example I would personally consider this "device" as busted. All of the numbers easily fall within the previously established margin of error and I would feel no particular need to continue testing. However, what if the results had been like this:

Example: Test vehicle in "B" condition
Run1: 45.5 mpg
Run2: 45.7 mpg
Run3: 44.9 mpg
Run4: 46.1 mpg

In this case every run is higher than the car in "A" condition. As a plus, it is even higher than our 1.2 mpg difference we previously calculated. At this point, the test cannot be considered conclusive at all. First, the test vehicle must be returned to its "A" condition and the runs must be repeated to make certain it returns to figures close to the original baseline. If it does not, then something changed the test conditions and the results must be scrapped.

If it does return to its baseline, the vehicle must now be placed back in "B" condition and the runs repeated once more. This is called "repeatability". If the numbers once again improve we know that the device is "working".

However, this still is not an absolute test. For one, only one vehicle is being tested. For another, the test should be performed again on a different day. Different vehicles and testing on different days would ensure that the device is indeed causing an improvement in mileage. If it really does work we should see slight to obvious results every time the device is added regardless of vehicle or environment.

[GasSavers_Jack]

Jeez I almost want to cut and paste that as a term paper.

Well done!

[95metro]

Thanks, Jack.

SVOBoy asked me to post this under Experiments to see if Matt would Sticky it. I think it's too short for a term paper...unless you add a legit experiment to it. Plus I don't know how it would look in a bibliography, "Testing method from some FE geek on the internet."

[Matt Timion]

proper testing is difficult, but consider the alternative: People reporting that devices work based on nothing more than "it seems to improve my gas mileage," or when people say ridiculous things like, "It improves fuel economy 8-20%." If they conducted an actual experiment why wouldn't they have an actual number?

I think real testing is what will seperate the men from the boys.

[95metro]

Hear, hear!

[cheapybob]

Your ABA method is correct, but short test runs can introduce error, too, IMO. Unless the temps, wind and road conditions are the same, they can skew the results, too.

I'm beginning to wonder if its really worthwhile. So few mods seem to make any significant repeatable difference. I seem stuck at about 53 mpg on my 60 mph test loop, and around 39 or 40 mpg for normal suburban driving. Both numbers are significantly better than what I started with, but additional improvements have all turned into mirages.

[GasSavers_Yoshi]

Just FYI...
Following is my collected data driving highway with CC set at 91km/h(actual was 85 km/h).

section km  mileage  alt change
    28km    24.5km/L      127m
    15km    18.7km/L      248m
    62km    40.8km/L     -663m

Saw such various mileage results even on constant speed.
The reason was the altitude change on each sections.
If we use 0.106L/100m fuel to cover the potential energy, the table becomes as follows;

section km  fuel used  fuel pot  fuel adj  mileage
    28km      1.14L     -0.13L    1.01L    27.8km/L
    15km      0.80L     -0.26L    0.54L    27.8km/L
    62km      1.52L      0.70L    2.22L    27.9km/L

Now, we can see constant about 27.8km/L mileage number for a virtual flat road.
Then, this number can be used for comparing the vehicle efficiency, such as tire change, Cd value change and so on.
And, the 0.106L/100m data can be used for comparing the engine efficiency, such as fuel change, acetone and so on.
Also, it shows calculated engine efficiency is 36% on my Prius.

Enjoy,
Yoshi

[Matt Timion]

Originally Posted by cheapybob

Your ABA method is correct, but short test runs can introduce error, too, IMO. Unless the temps, wind and road conditions are the same, they can skew the results, too.

I'm beginning to wonder if its really worthwhile. So few mods seem to make any significant repeatable difference. I seem stuck at about 53 mpg on my 60 mph test loop, and around 39 or 40 mpg for normal suburban driving. Both numbers are significantly better than what I started with, but additional improvements have all turned into mirages.

It can be a bit much to do testing. This is why I save testing for items I'm pretty sure are going to work. I'd rather be able to tell people "adding rear wheel skirts improved my fuel economy by 8%, and HERE is how I know."

Of course if an additive or gadget maker wants us to test for free I'd be more than happy to apply the same methodology to their product.

[95metro]

Originally Posted by cheapybob

Your ABA method is correct, but short test runs can introduce error, too, IMO. Unless the temps, wind and road conditions are the same, they can skew the results, too.

You're absolutely right, which is why establishing baseline and margin-of-error is so important, plus checking weather conditions for every run. You have to account for every possible condition and even then on-road testing is prone to error.

Long test tracks would be great, but unless you have a real race track to use weather conditions can change so drastically between points miles apart.

One thing I keep thinking as well is that every vehicle must have a plateau that can't be surmounted without extreme modifications (i.e. chopping your vehicle in half).

[Matt Timion]

Originally Posted by 95metro

One thing I keep thinking as well is that every vehicle must have a plateau that can't be surmounted without extreme modifications (i.e. chopping your vehicle in half).

I think you're right. And I think the plan is to find that plateau