One of the most popular — and in truth, most valid — ways of comparing two products is to, well, directly compare two products. A battle royale, two-enter-one-leaves style of head-to-head competition where it’s clear which product is the victor.
Done correctly, direct A/B comparisons are by far the most accurate ways of determining product superiority.
The problem is, they’re often not done correctly. Sometimes, they can’t be done correctly. In those cases, the results couldn’t be further from accurate.
Determining the best TV is one of the hottest Internet topics. It’s understandable, to a degree. Most TVs cost a lot of money, and most people want to be sure they're getting the best for their money. Then of course there’s the rampant fanboyism that is neither scientific, nor useful.
The problem is, it’s nearly impossible for the average person to accurately determine what TV is better. Not because of any failing on their part, or from lack of research effort. (Manufacturer specs are often outright lies, so they’re no help.)
The real problem is the lack of any decent testing space at retail. In a controlled environment, with correctly set up televisions, I’d bet good money that anyone reading this would pick the same TV I would. If not exactly the same, at least a TV that performs very similarly to my pick.
But you’ll never find a controlled environment. You’ll never see a correctly set up television. The Costcos and Best Buys of the world are about as bad a place to compare televisions as an Amish farmhouse.
The lighting in every store is radically different than what you have in your home. The plasma that looks washed out in a store will likely look better in your home than the LCD that looked amazing right next to it on the shelf. Worse yet, it’s unlikely the TVs at a store will be set even remotely correctly. They’re set to look great at a store, which is the exact mode you shouldn’t be using at home. I’ve even seen TVs in stores deliberately set brighter than others, because that was the TV they were pushing that week. Dishonest? Yes. Does it work? Absolutely.
A few months ago I wrote an article about why all HDMI cables are the same. In it, I outlined the physics of how HDMI cables worked, cited sources, and showed why your picture will look the same whether you use a $2 HDMI cable or a $400 HDMI cable.
I’ll admit to some sensationalism in the title (build quality is a big difference between cables, but it doesn’t affect picture or sound quality), but what was amazing was how many people said something along the lines of “well I have two cables, I compared them, I saw a difference, you’re wrong.”
Now, it’s not possible for them to have seen a difference. I proved that with, you know, facts. So what’s going on? The simplest (and most condescending) explanation is that they’re not seeing a difference at all. They want to see a difference to justify the $100+ by which they've just been ripped off.
Personally, I’m not a big fan of that answer. So let’s go a step further: how would one test cables at home? The most obvious solution, perhaps, is to plug in both HDMI cables to two different inputs on the TV and go back and forth. A direct A/B comparison, correct? Incorrect. Are all the settings the same between the two inputs? If there’s a visual difference, then probably not.
What about switching out the cables? Watch one, replace, watch the other. How long a time between light-on-eyeballs is there? No one’s memory is that good, especially with the added bias of knowing, “well here’s the expensive cable.”
Now this one is even more difficult. So difficult, in fact, it’s nearly impossible. Your ear is incredibly sensitive to changes in volume, and almost always louder is better. So matching volume is critical, but how do you do that? Most SPL meters aren’t as sensitive as your ears. Matching the volume on the receiver/preamp won’t work, as speakers all have different sensitivities, so “-20” on the receiver via speaker A might be louder than the same level through speaker B.
Then there’s placement, as the speakers need to be in the same spot so the room has the same effect on them. Everyone’s audio memory is pretty fickle, so if you wait around while someone moves the speakers into place, it’s likely you’ve forgotten what the speakers sounded like to begin with.
Is this a thinly veiled laudation of magazine reviews and trained reviewers? Sort of (well, at least the good ones). I’ve always said that you should trust your own eyes and ears when it comes to audio and video gear, and let them be the final judge. But let your knowledge of these potential errors in direct comparison be your guide. With any test, isolating the variables that can affect the outcome, but aren’t themselves the factors under test, is more important than the test itself. This is true even with A/V gear.
With TVs, keep in mind that a well-lit store isn’t the best place to compare TVs and that the TV’s settings will have a profound effect on the picture quality. If you can’t access them, it’s nearly impossible to correctly judge the display.
With cables (or any piece of gear, really), know that the simplest A/B test may have significant variables that can dramatically affect the outcome. When the results go against physics — well, I don’t know what to tell you. Take a close look at your test setup. There may be a difference somewhere, and it's not in the cable.
With speakers, trust your ears. Whatever sounds good to you is the best speaker.
As any good scientist will tell you: If you’re getting results radically different from everyone else, it’s probably time to re-check your testing methodology. Sure, it’s possible you’ve discovered penicillin, but it's more likely you’ve just got a jar of mold.
Are you trying to say there is no difference between speakers? I totally agree about the cables as long as they are built well there is no difference but speakers? You are forgetting about people who know nothing about audio equipment and will choose by the price over how they sound. I will agree there is not that much difference between speakers in the 500 dollar range and the 2000 dollar range but come on now. Lets get real.
" isolating the variables that can affect the outcome, but aren’t themselves the factors under test, is more important than the test itself. This is true even with A/V gear". Huh? What the hell does that mean?
"isolating the variables that can affect the outcome, but aren’t themselves the factors under test, is more important than the test itself. This is true even with A/V gear"
It means you're trying to remove those variables that you aren't testing, such as room size, volume, and so on from the test, as they can affect the outcome, but they aren't what you are actually testing.
If you've read Floyd Toole's book on sound reproduction, you can understand how hard it is to test speakers. He writes that most things like break-in aren't really physical things occuring, but your mind adapting to a sound over a period of time and learning to live with any issues that are coming up. He also goes into depth about the listening room that Harman built to A/B test speakers, that allows them to be volume matched, hidden from the listener, and moved into the exact same position as another speaker at the touch of a button. It's an incredibly well designed setup, but one that is far beyond the realm of any normal reviewer.
Asking a reviewer to compare products they have months apart is a challenge. They can review their notes for what they thought, but once a product has left it's often very hard to remember specifics compared directly to another product unless you still have it around. Even then it's a challenge since you can't easily swap them usually, as Geoff said.
@chris I had a section in the article about Harman's A/B test rig, but I took it out for various reasons. Their rig is amazing, and probably the most accurate way to compare speakers.
@koufax I'm not saying there's no difference in speakers, just that the final judge should be one's own ears... if possible. It's pretty hard to find a place to demo speakers these days.
Speakers, calibrating speakers is a personal preference. I still have clients that want to always here things out of the rear speakers; they do not understand that the only thing that should come out of them is things that are happening behind you. They think you are supposed to hear the news in surround. We usually set it up as we would at home and then show them how to adjust them as to their personal preference. As for the “With speakers, trust your ears. Whatever sounds good to you is the best speaker.” is not true. You cannot go to Wal-Mart or Best Buy and listen to a speaker system and then say it is as good as a pair of B&W Nautilus speakers, not that I would spend eighty grand on a pair of speakers. Most of our client’s speaker choices are based on budget. What I hear a lot is “they are good enough for me”. But as one of the comment below says that between 500 and 2000 dollars there is no difference, you have to be kidding me. Between bandwidth, frequency response and woofer size not to mention the crossovers, there is a difference between a 500 and 2000 dollar system.
http://www.unisenmedia.com/ ">Unisen Media LLC
Cables, there is a difference between a $2 & $100 cable. Now does it suit everyone, no. If you connect a lower brand of TV or DVD/Blu Ray it won’t make a whole lot of difference. If you connect a 480htz 3D TV with a good quality 3D Blu Ray it will. Try it; the difference in refresh rate, smoothness of motion & pixilating. Doesn’t mean that you have to spend a $100 on a 6ft cable, but a good one will run 10-45. It baffles me why people will spend $5000 on a TV and buy there cables from blankprice.com. As for 75ft runs higher quality does matter. You do not have to buy Monster 75ft cables @ $400 but paying 100-150 for a 75ft is about right. Also be sure if you buy from a custom installer he is not selling you that $2 cable for 10-45. Remember they buy from blankprice.com to. Now back to the $2 or so cables. With lower price cables the quality comes in to play. Let’s say you have a PS3 & you move it between rooms. A $2 cable is not going to last you as long as a 10-45. If they even twist a little the heads pop off. If that happens then not only is the cable useless but if they TV is on there is a chance it will blow the HDMI board. Also if you have a TV mounted on wall or a projector & screen & the HDMI line is run through the walls, if it goes out then you will incur the cost of having a new one installed. Cheaper is not always better & on the other had the most expensive is not always better but they both have their place & uses. http://www.unisenmedia.com
@unisenmedia I agree that "they are good enough for me" is a cop out. I've written before about how many people feel that they won't be able to hear a difference between speakers, so don't bother even trying. What I found in my sales days was that everyone can hear a difference, if given the opportunity. The problem is, there are very few places to have that opportunity. There are fantastic sounding speakers at many different price points. I've heard excellent $500 speakers, mediocre $2,000 speakers, and horrendous $80,000 speakers (and vice versa).
My point is, people should trust their ears more.
Your take on HDMI cables, though, is incorrect. It is not physically possible for an HDMI cable to have any affect on refresh rate, smoothness, pixelating or any other artifact other than sparkles or a total lack of picture. I recommend checking out the linked article in the cable section. A $3 HDMI cable will look the same as a $400 HDMI cable. As I mention in the article, build quality is a differentiator, but for most people that's unimportant. It will not make your TV look better, regardless of price. It's not possible with the HDMI technology.
If someone is running the cable through walls then yes, a better made cable is advisable, but "better made" often has little to do with price.
Your statement about HDMI cables all being the same is DEEPLY flawed! You CAN and WILL see picture issues if using a substandard cable! Errors in transmission result in error correction being used by the display (or video processor). "Missing" or "wrong" information is run through the error correction and an APPROXIMATION of the original signal is displayed. This results in a softer picture, or poor color rendition, and it CAN be seen!
This is also an issue when using Cat5 extender products. Poor ones (most dual cat5 solutions) will NOT pass the HDMI signal 100% without errors! I have PERSONALLY seen a difference when using a poor extender (sometimes referred to as a "balun", although technically this term is incorrect), and MEASURED the errors with a Quantum HDMI certification tool.
Go out and get your own Quantum tool (they are about $10,000), test some cheap cables, then come back and tell me how HDMI cables are "all the same"!
I like your website and magazine, but I am SHOCKED at the quality of your journalism here! Especially since so many rely on good advice, and articles that state NONSENSE like this are doing an INCREDIBLE DISSERVICE to your audience!
@jnemesh: The test results of cables on an analyzer are irrelevant here. The fact that you can measure performance differences among cables doesn't prove that such differences result in visible changes in the image. If one can actually see these differences, cable manufacturers would be eager to demonstrate them. To my knowledge, they haven't.
I'm confident I can speak for Geoff when I say we invite any HDMI cable manufacturer to demonstrate the visible differences its cable produces when compared with a good generic cable. I'm equally confident that no manufacturer will take us up on that offer.
@brentbutterworth The tests are EXTREMELY relevant! If you put a signal through a cable and the signal coming out the other end doesnt match what you put into it, you are degrading the signal! If the errors are plentiful enough, YOU WILL SEE THE DIFFERENCE! Error correction circuits can only do so much! Take a 50 foot cheap cable that you find online and compare it to a brand name cable. Use a projector or a 60" + flat panel so you can see the differences clearly, and you will notice a difference! I HAVE SEEN THIS MYSELF...and despite what you have read elsewhere, there IS a difference!
If there were no difference, as you claim, then there wouldnt be any measurable difference that you could see with the HDMI certification tool or your own eyes...in both cases you CAN measure and SEE a difference. If you think otherwise, try it yourself. But honestly, I am VERY tired of the misinformation on the subject that is being perpetuated by IGNORANT articles like these! If there were no difference, why are there products on the market? Do you think EVERYONE is just selling snake oil? Not likely!
OK, enough. My "ignorant" article is backed up by cited sources, data, facts, science, research, and detailed conversations with multiple cable manufacturers, HDMI Licensing (who write the actual HDMI spec) and Analog Devices (who make the sending and receiving chips). The article is a distillation of several months of research into how the technology actually works, culled from multiple reputable source, and my own testing.
As it is clear you didn't bother reading the article (out of necessity it's long, I know) let me go over the facts:
It is possible, as you say, that different cables are more or less likely to transmit the data. A better made cable, over long runs, will be more likely carry the data from source to display.
Where you are incorrect, is in the nature of failure if the entire signal doesn't get to the display. It is not possible for the cable to *change* the data. Each pixel either arrives at the display exactly as it left the source, or it doesn't. If it doesn't, that results in a dead pixel. I have pictures of this type of failure in the CNET article. It's called "sparkles" and looks like snow.
(cont...)
(cont...)
Think of it like a train. Each car on the train is a pixel. If the train pulls into the station and a car is missing, the station doesn't just "make up" a new car. It can't. The train also doesn't get uglier or shrink its travels. It either arrives perfect, or it doesn't arrive. It's 1s and 0s, there are no partials.
There is also NO error correction on the video portion of the HDMI signal. TMDS is robust enough that it effectively doesn't need it for nearly all uses. The only error correction is on the Data Island Period, which carries all the audio and additional information like HDCP.
Unlike analog, there is no correlation between the signal quality and the resulting picture quality. It remains perfect until it doesn't. It doesn't get progressively worse, it just stops being a picture (I'm including sparkles in that).
As for what you have seen, well, that's exactly what this article was about. Thank you for proving my point.
@jnemesh: If you spent $10K on an HDMI certification tool, I assume you are a cable manufacturer. So the solution here is simple: Demonstrate the difference to us. My contact info is at the bottom of my website, brentbutterworth.com.