[ The PC Guide | The
PC Buyer's Guide | Designing and Specifying PC Systems and
Components | Key Performance Issues In PC System Design ]
Using Review Resources Wisely: Avoiding "Benchmark Bogosity"
When you go to shop for components and systems you will be able to take advantage of
numerous resources that exist to help you assess hardware, manufacturers and vendors. I discuss these in some detail here. Many of the research
resources that evaluate hardware components and systems include as part of their analysis benchmarks.
A benchmark is a measurement that is intended to help indicate the performance level of a
component, subsystem or system in a standardized way, so that comparisons can be made
between similar hardware models, and help you decide what to buy.
Benchmarks are a useful indicator of performance--when used properly. When used
improperly they are worse than useless, because they become deceptive and can lead to bad
decision-making. When you see benchmarks stated in an online review, or mentioned in a
manufacturer's literature, be sure to take them in the proper context. Always remember
that many benchmark numbers--and the conclusions some reviewers and manufacturers draw
from them--are highly suspect.
Here are some specific tips to keep in mind when considering hardware benchmarks of any
sort, when reading a review or product literature:
- System Benchmarks: Benchmarks are rarely run on entire systems, because the
results vary based on the exact components in the system, and because the numbers aren't
readily comparable between different systems. Be wary of any company that tries to boil
the performance of an entire PC down to a single number. Unless you have that exact model
the number doesn't tell you much.
- Component Benchmarks: Most benchmarks are usually run on specific components,
such as CPUs, hard drives and so on. These usually have some meaning; if CPU #1 benchmarks
faster than CPU #2, it will likely result in some sort of performance improvement
in your machine, if you change from #1 to #2. However, the improvement can vary greatly,
depending on the other hardware in your machine. It is impossible to completely isolate
the performance of one component from that of the others. If the system that the test was
done on is very different from yours, you should not expect to see the same performance
difference that the reviewers did when they compared #1 and #2.
- What Is Being Measured: Find out all the details about any benchmark before
deciding to take the results it gives seriously. What exactly is being measured, and how?
- Relevance: A particular component may produce a benchmark that is 20% higher than
the benchmark value of another component, but how relevant is that result? Each component
only contributes to one aspect of overall performance, so these numbers do not translate
directly in the real world. A component that is 20% faster than another will not result in
the PC as a whole being 20% faster. It could be only 5% faster. In some cases it will make
no noticeable difference at all.
- Significance: You'll occasionally see a bunch of components of the same type
compared, and the benchmarks scores of the units come out very close together. This
commonly happens with motherboards that are put into systems that otherwise use identical
hardware: there might be four boards benchmarked, with the fastest board 5% or less higher
than the slowest. Sometimes a graph is displayed with the zero point cut off to
"magnify" the relative differences in scores between the different devices. In
such a case, even if the benchmark scores are valid, they are not significant. A
general rule of thumb says that most users won't even notice a difference in
overall performance of less than about 10%. If motherboard A is 2% faster than motherboard
B, then the two can be considered roughly equivalent in terms of performance. (Heck, the
margin of error in the benchmark program itself is probably higher than 2%.)
- Proper Emphasis: Some reviews of hardware seem to focus almost entirely on
performance benchmarks, providing page after page after page of graphs and tables. This
represents plenty of data, but is any of it useful information? Benchmarks
are often emphasized because they are easy to do: it is not hard to run a benchmark
program and create tables from the results. It is much more difficult to assess what those
results mean in a way that makes sense, or to evaluate the quality, reliability and other
characteristics of a product.
By far the most important thing to look for in a review that uses benchmarks is the methodology
used by the reviewers. Are they using the benchmark properly? Are they running each
benchmark multiple times and averaging the results to reduce the chances of spurious
numbers? Are they remembering to change only one variable at a time? This last one is
important: the only way to even have a chance at comparing the performance of two
different components of the same type is to use them in the identical system and test them
under identical conditions.
The bottom line? Take benchmarks with a grain of salt. Use them, but keep them in their
place, and don't succumb to the temptation to assign them more value than they are worth.
Tip: For more on benchmarking, see this article.
Next: PC System Balance
Home - Search
- Topics - Up