Learn about the technologies behind the Internet with The TCP/IP Guide!
NOTE: Using robot software to mass-download the site degrades the server and is prohibited. See here for more.
Find The PC Guide helpful? Please consider a donation to The PC Guide Tip Jar. Visa/MC/Paypal accepted.
View over 750 of my fine art photos any time for free at DesktopScenes.com!

[ The PC Guide | Systems and Components Reference Guide | Hard Disk Drives | Hard Disk Performance, Quality and Reliability | Hard Disk Performance | Hard Disk Performance Measurement ]

Objective Performance Measurement (Benchmarking)

Benchmarking hardware is a form of objective performance measurement; it is measurement based on logic and analysis, as opposed to subjective measurement, which is more of a "feel" method of gauging performance. Benchmarking is typically done using benchmark programs specifically developed for the purpose of measuring hard disk performance.

There are many different programs used to ways to benchmark hard drives, and they generally fall into the following different categories:

  • High-Level (Application-Derived) Benchmarks: These are programs that use code from popular applications software--usually office applications, web browsers and the like--to simulate the impact of hard disk performance on the use of those applications in the real world. The basic idea is to run a suite of tests that are comprised of typical actions that a user would take while using those applications, and time how much time elapses with the hardware in question. Then the hardware is changed and the test run again. This is generally a good concept for benchmarking, but only has relevance if you are actually using the types of applications around which the benchmarks is designed. If you are primarily a gamer for example, what do you care about the performance of spreadsheet software? Also, since the benchmark is running at a high level, there is a lot of room for interference from operating system and file system issues. One of the most common benchmarks of this type is the ZDNet WinBench series.
  • Low-Level (Synthetic) Benchmarks: These programs attempt to test the hard disk directly, isolating it as much as possible from the rest of the system. They are often called "synthetic" because they don't try to reproduce the access patterns of real applications, instead using artificial patterns created by the programmers specifically for the benchmark, to test different types of hard disk use. They are often derided as being unrealistic because of their synthetic nature, and much of this criticism is in my opinion accurate. At the same time however, they provide much more control over the test process than application-derived benchmarks. This control lets you better "focus in" on one particular aspect of performance and more accurately compare different hardware units in a number of different areas. Common disk benchmarks of this variety include Adaptec's Threadmark and Intel's IOMeter.
  • "Real-World" Benchmarks: These are not "formally" benchmarks, but are commonly used by hardware enthusiasts to compare real-world performance of hard disks. The idea is simple: take something that you do often, measure how long it takes with one drive, and then how long it takes with another. For example, if you have a system with an old hard disk that is very slow to boot up, measure how long it takes and then repeat the process with a newer disk to see how much things improve. In some ways these are the most realistic, and also the most relevant benchmarks. However, they are entirely system-dependent and therefore of no use whatsoever in communicating much in objective terms about the power of the hardware in question: the improvement you see between hard disk "A" and hard disk "B" on your system may be very different than the same hardware used on a friend's PC. Also, these measurements are usually fairly crude and can't be done on activities that take relatively little time, since the timing is often done with a regular clock or wristwatch.

As I've said elsewhere, I'm not a big fan of benchmarks, especially when it comes to hard disks. While they have their uses, it's too easy to succumb to the temptation to view them as absolute indicators of performance, to overvalue them and not consider what they really mean, bottom line, for the typical end user. Small differences in hard disk performance have virtually no impact on the typical hard disk user. Some people really get carried away, sweating over every percentage point of their favorite benchmark, as if it were a competition of some sort (and for some people, I suppose it is--a competition for bragging rights.) Even leaving the matter of over-emphasizing benchmarks aside, there are some common "benchmark traps" I see all the time:

  • Poor Control Of Environmental Factors: The only way to properly compare two pieces of hardware is to test them under identical conditions. Even seemingly irrelevant issues can influence the outcome. Most better hardware sites understand this, but many individual enthusiasts do not. The exact number you get from testing one drive on your system can be very different from the number someone else gets with the same drive, without this meaning anything is "wrong".
  • Small Sample Size: All benchmarks have a tendency to produce different numbers if you run them more than once. To properly use a benchmark it must be run several times and the results averaged. It's even better to run at least five times and discard both the highest and lowest score for each piece of hardware.
  • Paying No Attention To Cost: You will frequently see people talk about the "benchmark X" score of one drive versus another, but when's the last time you saw anyone take the ratio of two drives' respective benchmarks to their current market prices? I've seen people recommend "drive A" over "drive B" due to a difference in performance of well under 10% despite "drive A" costing 50% more than "drive B". That's rarely money well-spent.
  • Benchmark (In)Validity: It's not uncommon to see a particular benchmark be used for a long time by many people... and then it is discovered that due to a flaw in how it is written, or the manner in which it interacts with the hardware, operating system or drivers, that its results were inaccurate or misleading. Another reason to use benchmarks only as guidelines.

Next: Subjective Performance Measurement

Home  -  Search  -  Topics  -  Up

The PC Guide (http://www.PCGuide.com)
Site Version: 2.2.0 - Version Date: April 17, 2001
Copyright 1997-2004 Charles M. Kozierok. All Rights Reserved.

Not responsible for any loss resulting from the use of this site.
Please read the Site Guide before using this material.
Custom Search