A few weeks ago I found myself mentioned in a rant by CNET's Steve Guttenberg. Steve thinks it’s dumb for anyone but a product designer to measure the performance of audio gear. He mentioned me because I take the opposing view.
Steve made some good points in his blog, citing instances in which trusting too much in measurements has led the audio industry astray. But IMHO, he overlooked the many valid reasons for reviewers to measure audio gear as well as listen to it. I guess the job of filling in the missing half of his article falls to me, so here goes. . .
I’ve spent 22 years covering the audio industry, as a senior editor at Video magazine, editor-in-chief of Home Theater and Home Entertainment, and contributing technical editor of Sound + Vision. I’ve known countless reviewers, from some of the guys who started the whole thing back in the 1960s to writers in their 20s and 30s who’ve been at it for just a year or two. I’ve hired many dozens of them, edited their work, and checked out the products they’ve reviewed on my own. And after all that experience, there’s one thing I’ve learned:
I don’t trust them. Not 100 percent, anyway.
Even the best and brightest of reviewers can overlook serious flaws in a product. Any of us can fall in love with a piece of gear for the wrong reasons. Maybe the device does one thing so spectacularly well, we inadvertently ignore a problem that a user with different priorities would find in minutes. And as much as every reviewer tries to be unbiased, we all expect certain brands to be good and others to suck. We’re human. We can be prejudiced. We can be swayed.
My measurement gear, on the other hand, cannot be swayed. A manufacturer can’t buy lunch for my Clio FW and Audio Precision analyzers. Test gear doesn’t care how much a product costs or how cool it looks or how hot the public-relations person is. It just tells you the facts.
Here’s an exercise that illustrates my point: Look in any high-end audio magazine and you’ll see rave reviews of esoteric, expensive speakers. Go hear a few of those speakers at a dealer or a hi-fi show. I guarantee you will hear one or two that blow you away with sound quality you didn’t think possible. I also guarantee you’ll hear one or two that leave you questioning the reviewer’s sanity and/or integrity.
And I absolutely, positively guarantee you that the speakers you don’t like don’t measure well.
In fact, of all the countless speakers Steve reviewed for me when I was an editor, there were only a couple that he loved that I didn’t like. The frequency response measurements of those speakers were horrible.
Those speakers’ unusual properties clearly seduced Steve, and it’s important that we hear that side of the story. But it’s equally important to be aware of technical shortcomings that a subjective reviewer might miss. These speakers’ flaws apparently didn’t cause problems in Steve’s listening room, or with the music he used to test the speakers. Or maybe his taste in sound allows him to tolerate these speakers’ particular problems. But your tastes in sound and music likely differ from Steve’s, and the acoustics of your listening room definitely differ from his.
Those of us who measure audio gear as well as listen to it don’t blindly trust our meters. We trust science. Many years ago, researchers at Canada’s National Research Council in Ottawa decided to find out if one could use measurements to predict which speakers listeners would like. They found out that indeed, you could. In blind tests, listeners tend to prefer speakers with a frequency response that’s flat on-axis and smooth off-axis, with no major peaks and dips until you get way off-axis.
Such speakers also tend to perform more consistently in a variety of acoustic environments, i.e., they’re not as affected by room acoustics. This isn’t opinion. It’s scientific fact based on countless listening tests, all of which were far more meticulous than those that reviewers conduct in their homes.
I challenge you: Name a speaker that delivers flat on-axis response and smooth off-axis response that doesn’t sound good. I’ve reviewed and measured way more than a thousand speakers, and I can’t think of a single example.
Measurements also allow the reviewer to gauge a product’s suitability for other listeners’ tastes and listening rooms. In a recent review, I compared the Hsu Research VTF-15H subwoofer to the SVS PB12-NSD. Even in my fairly large, 2,950-cubic-foot listening room, playing DVDs and Blu-rays at full Dolby reference level, I could barely tell the difference between these two subs. But my measurements showed a clear advantage for the larger, more expensive VTF-15H in the bottom octave of bass (20-31.5 Hz). If you have a larger room or want to play at even higher levels, or if you really love super-deep bass, my subjective evaluation would have told you absolutely nothing about these subs. My measurements, however, tell you everything you need to know.
None of this means that the speaker that measures better always sounds better. About six years ago, I tested two speakers with similar form factors: the Earthquake Sound Platine Noirée and a tower speaker from Definitive Technology’s Mythos line. (The model number escapes me.) The Mythos measured great and sounded great; they were the product of meticulous, by-the-book engineering. The Platine Noirée measured badly, with major peaks and dips in the frequency response. It was obvious when I disassembled the Platine Noirée that it wasn’t the product of a careful engineering effort. Yet I preferred its sound.
This could have been for any number of reasons. Maybe my room’s acoustics suited the Platine Noirée better. Maybe its sound was particularly appropriate for my taste or the music I listen to. I don’t know, but I gave the readers both my subjective impression and the measurements, so they’d know the pros and cons and they’d be sure to give the Platine Noirée a good, long listen before purchasing it.
I’m not saying reviewers should ignore their ears. Ultimately, any audio product must be judged on whether or not the sound arriving at your ears pleases you. But a single reviewer’s ears cannot provide a complete assessment of the performance of an audio product. The only way to produce audio equipment reviews that are useful for any and every reader is to combine the art and the science, the subjective and the objective, the ears and the meters.
It all comes down to human error and human error also has bias. No matter what anyone says people have bias, we can not help it. Whether it be due to a single thing that is good that creates that bias, nostalgia, price, money or the way something looks. I remember when boom boxes were big and they went from good quality sound systems (for a box with speaker) to how many lights does it have. Also how do you trust a review of X when X takes out a full page ad in the same magazine or banner on their home page. Do you think anyone should trust that review. Do they really think that I believe that they are going to trash a product that is paying them money to place that ad. If it was my product I would pull the ad from anyone that trashed it. Keep testing, consumers need all the reviews they can get from product designers, reviewers and consumers themselves. In the end it is a lot to get through between the brands ads, fake reviews produced by those brands, product designers with financial motives, magazines with financial motives and the stores whether they be physical or online that sell those brands and push them. It seems now a days that the only thing for consumers can do it buy a product take it home try it out and decide for themselves, this has been made possible by all the 30 day return policy's. www.unisenmedia.com
Without measurements readers will not know what a product is capable of whether the reviewer could hear it or not. Listening room, mood at the time, preconceived notions about a piece of equipment all affect a review. Measurements also can debunk an over optimistic spec sheet provided by the manufacturer (an all too common occurrence). I consider them to be an integral part of the big picture before auditioning a component myself.
Hi good to see healthy debate at last. I once attended a dealer only show where a turntable was fed into a AD/DA converter and then to a pair of active speakers. The switching was random and done a total of 20 times.
Using our ears about 50 or so high end dealers tried to identify whether the music had been digitised or not. This was in the 80s and the digital vs analogue debate was raging!.
The result well out of the 50 who tried 2 achieved a statistically significant result Chris Brooks managed to get it wrong 19 times and I managed to get it right 20 times.
Strangely when we first tried it I could not detect any changes however when the turntable (direct drive) was substituted with a belt drive LP12 it to me anyway became easy.
But which sounded better? no idea the enjoyment of music depends on so many variables that it is impossible to build a once and for all reference. Thank goodness or we would all end up listening to an ipod plugged into a bose dock!!!!!
Long live subjectivity and long live objective measurement both are needed to keep our hobby so involving and fresh. I am off now to play my old speakers while I still have ears that work.
Part 1. Why we measure
I've been involved on both ends of the spectrum: measuring and listening. I have never failed to dislike a speaker that measured well, but I have often disliked speakers that measured poorly and price wasn't considered. In the 1970s I tested a tower loudspeaker (4-way, 1st-order crossovers with the impedance characteristics of the drivers considered in the cross-over design) that sounded good, but not great in spite of the coherence of the sound. The problem was due to ripples in the mid- and high-treble caused by standing waves set up across the speaker's baffle that acted as a comb filter, if you will. Since rounding the baffle wasn't an option, applying a thin sheet of open-cell foam on the baffle surface solved the problem. The on-axis response was flatted in the troublesome range and the speaker sounded spectacular. It's response was from 22 Hz to 20 kHz +/- 1.5 dB on axis anechoic. The design was later sold to a larger speaker company and it ruled the roost in what passed for high-end audio back then.
Part 2. Why we measure
The human ear, and its attached attentive listener, is sensitive to irregularities in on-axis response and the smoothness of the off-axis response for almost any speaker. With the advent of smaller speakers has arrived smaller baffles and better management of standing waves inside and on the surface of the enclosures. Those speakers that reviewers say "disappear" or "cast a sound stage that is wider than the physical distance" are usually referring to speakers with flat on-axis responses and smooth off-axis responses. So, while a good review accompanied with excellent measurements tells me something about the speaker (witness this year's GoldenEars), a good review accompanied by NO measurements tells me nothing. I often flip to the measurements section of a review first before I read the review, or as in the case of Stereophile, I read the technical section right along with the review. And, I wish S&V could turn back to the day of more expanded reviews somehow.
By the way, Gutenberg and I have similar backgrounds to a point: Ex-movie theater projectionists and high-end audio salesmen. There we separate because I made audition, physics, electronics, mechanical engineering, audiology, and scientific research (all of those things can can be lumped into bio- and psychoacoustics) into a career, rather than technical writing,
I'm glad that a few people, including myself are still on the same page. Most important of all, it's a relief to see that a professional equipment reviewer such as Brent has the guts to argue for objectivity.
For a while, I thought objectivity was dead and I was the only person on the planet supporting it. We're living in an age where most so-called professional reviewers and manufacturers are belching words and pushing snake oil.
I can still remember one subjective reviewer, currently (brace yourself) an editor in chief for a major subjective publication, who once blatantly attacked the practice of the traditional ABX testing that was used to disprove the notion of apparently obvious differences between speaker cables and interconnects. He actually suggested that the test itself should be tested!
I'm totally with Brent on balancing objective measurements and analysis with subjective commentary. In fact, I have always valued most of the past reviewers in the early Audio, High Fidelity, and Stereo Review equipment reviews for their practice of measuring component performance then correlating their impressions with the measurements, whenever possible. I find pure subjective evaluation to be virtually pointless, as it only reveals little more than bias.
Hopefully, some of the younger budding audiophiles will acquire some common sense out the current frenzy of delusional listeners and reviewers.