Some readers shy away from the “in the lab” boxes in our test reports, probably because it’s hard to judge what represents desirable performance if you don’t have a lot of experience with the kinds of figures we publish. To help you interpret how our numbers relate to what you see and hear, we’re going to lead you through the data in a series of articles discussing each of the major equipment categories, starting with DVD players.
For simplicity, assume that we’ve tested a DVD player equipped to play the new multichannel DVD-Audio music discs. Instead of actual test results, the “in the lab” box printed on the facing page shows the ideal result for each parameter. How to evaluate a real player’s measured deviations from theoretical perfection, or zero error, makes up the rest of the story. First, though, let’s take a look at how our test-data boxes are organized in general.
The Layout
Each “in the lab” box is separated into sections covering the tested
component’s performance in its major operating modes. Here it is a player’s
video performance with DVD-Video discs and its audio performance with DVD-Audio
discs and CDs. We’ll discuss Dolby Digital performance, which can make
up a large part of some players’ test data, later in this series when we
cover lab data for A/V receivers and amplifiers, where it’s more relevant
to a purchase decision. (It’s better to have Dolby Digital decoding done
at the end of the signal chain, where it can serve other source components,
such as a satellite receiver or a high-definition TV tuner, in addition to the
DVD player, and where the bass-management facilities may be better.)
Each section starts with a list of significant test conditions, which are chosen to represent typical equipment settings and typical input signals. We list these because they sometimes differ from “standard” industry practice, especially with receivers and amplifiers. Fortunately, few user settings on a typical DVD player affect measured performance, and we normally test a player with its default, straight-out-of-the-box settings. At the end of each lab box we print comments by the tester, who will usually discuss any particularly surprising or unusual results or put the findings of key tests in perspective.
DVD-Video Performance
Unlike the multiple audio measurements that appear in a player test, the
video results are straightforward. The reason we use a composite-video connection
for most of these measurements, by the way, is that it typically represents
the worst-case output. S-video measurements, when applicable, will almost always
be equal or better, so if the player performs well with composite connections,
you can assume it will perform well with S-video, too.
Maximum white-level error indicates any errors in overall picture brightness that might need correction using your TV’s brightness controls. Many players come out of the box with a video output voltage that is too “hot” (high), and they can look slightly brighter than players with no maximum white-level error, measured in IRE (1 IRE is 1/100 of the ideal dynamic range of the luminance, or brightness, portion of a video signal). If you compare two DVD players that have significantly different maximum white levels (a disparity of 3 or 4 IRE) without compensating for the difference, you’ll probably prefer the one that produces a brighter picture, all else being equal, even though it may be a less accurate image.
At the opposite end of the video dynamic range, setup level indicates what IRE setting the player uses to represent black. The level for the direct-view monitors used in TV studios and during DVD mastering is normally +7.5 IRE, and setup discs like Ovation Software’s Avia also aim for that level. Any monitor/player mismatch here — specifically, the use of an “expanded” or “enhanced” black-level setting of 0 IRE in the player — will throw off the brightness scale on a TV adjusted for the +7.5-IRE standard. While this has the effect of increasing contrast, which many people find appealing, it also reduces the amount of detail visible in dark images or deep shadows.
Differential gain and differential phase are two traditional composite-video measurements that are helpful in evaluating videotape recorders but are not especially relevant to the quality of a DVD player. We continue to run them for comparison with other devices. Differential phase indicates how much the color shifts as the point-to-point brightness of the picture changes, and differential gain indicates how much the brightness of the picture changes as the color shifts. Unless there’s a direct comparison with a reference image, the eye is extremely tolerant of larger errors in both these parameters, especially differential gain, than we have ever measured from a DVD player.
Luminance frequency response (formerly called “horizontal luminance frequency response”) is the best way to judge a DVD player’s resolution performance. This measurement allows you to decide whether the picture detail you’re seeing comes from accurate playback (a flat response with ±0-dB error up to 6.75 MHz) or from a “goosed” response (+0.5 dB or more between 4 and 6 MHz), which can make a picture seem sharper without necessarily conveying very fine detail. Better players will have flatter response up to a higher frequency than one that rolls off or has a “midrange” boost.
If you’re deciding between a player that rolls off slowly at the higher frequencies (say, –1 dB at 4 MHz, –3 dB at 5 MHz, and –8 dB at 6.75 MHz) and one that is flat throughout the range but down 10 dB or more at 6.75 MHz (the DVD-Video format’s limit), choose the latter. A player that has flat response will provide a sharper-looking picture than one that rolls off.
To
cross check our luminance-response measurements, and to give readers a resolution
figure with the more familiar “lines of resolution” as the units,
we use onscreen horizontal resolution (formerly just “onscreen resolution”).
It’s one of two DVD player tests we make (the other is in-player letterboxing)
that are judgment calls, meaning that the numbers aren’t spit out by a
machine. Each requires looking at a test pattern on a TV screen and deciding
what the result is for that test. The test pattern in this case is the 200-line
resolution “wedge” chart of the Avia test DVD. The area we look at
is the bottom of the righthand vertical wedge, and what we look for is the point
where the closely spaced vertical lines turn to a gray mush, with no vertical
line structure visible. That point, if it occurs at all before the bottom of
the wedge is reached, is converted to a resolution measurement by reading off
the scale next to the grayed-out point of the wedge.
Nearly all DVD players have enough output at the format’s resolution limit of 540 lines (equivalent to the 6.75-MHz limit for luminance frequency response) to merit a 540-line reading for onscreen horizontal resolution. The few that don’t still might not look obviously “soft” with movies, because their response at lower frequencies is flat or boosted and because the amount of fine detail that benefits from full 540-line resolution is small.
Pixel
cropping, measured using a test pattern from Avia , is important only if your
TV can show a full video image without “overscan,” since it indicates
where and by how many pixels a player cuts off the edges of the image data as
it is converted into a video signal. Not many TVs meet this criterion —
mainly some projection sets and some widescreen TVs that show standard 4:3 aspect
ratio fare with letterboxing bars on the sides (DVD playback even over a computer
monitor can introduce pixel cropping). If you have that kind of a set and want
to see the entire recorded frame, the less pixel cropping in the player, the
better. The losses are almost always trivial, but hard-core videophiles get
quite worked up about them.
In-player letterboxing indicates the visible quality of a player’s conversion of an anamorphic or “enhanced for widescreen” DVD for display on a standard 4:3 screen, so it’s relevant only if you watch on a 4:3 screen. This is the single most subjective “measurement” we make of a DVD player, but also the one where differences between players are most easily visible.
Letterboxing is a mathematical transformation of the video data, similar in principle to an audio sampling-rate conversion. Depending on what equations are used, various artifacts can be generated on images that contain closely spaced horizontally oriented patterns, like thin Venetian blinds, or tight patterns that move through the horizontal plane, as in the “test pattern” we use — the polka-dot shirt Mike Meyers wears in Chapter 8 of Austin Powers, International Man of Mystery. With players that have “poor” letterboxing, the shirt will appear both dotted and striped as Meyers moves around because the letterboxing artifacts distort the dots so they run together. “Good” in-player letterboxing will consistently produce dots only. “Fair” players fall somewhere in between, with faint striping visible at times. Most players have clearly poor performance in this test, the tough call being between the few that merit a good or fair rating.
Component-output level error indicates mistracking between color and brightness, though with component video these are conveyed as three distinct signals (luminance and two color-difference signals). Large errors here can result in color and brightness shifts that may not be correctable using a screen’s picture controls. A ballpark figure for “good” performance is any value within ±5%.
High values of component-output timing error (greater than ±10 nanoseconds) can lead to color-fringing effects on hard-edged objects in an image. Smaller error values are not generally visible.
Audio Performance
We measure the same basic parameters for DVD-Audio and CD playback. For space
reasons, CD measurements are often omitted in our published reports, but if
there’s anything unusual about them, whether positive or negative, it’ll
be mentioned in the notes following the measurements. Keep in mind that if you
use a player’s digital audio output only, feeding its signals (including
Dolby Digital and DTS soundtracks) to a digital surround receiver or preamplifier/
processor, measurements of its CD and (if applicable) Dolby Digital or DTS performance
become irrelevant to a purchase decision, with the sole exception of CD defect
tracking.
Maximum output level is also not something to base a purchase on. We measure it and publish the results on the off chance that you’ll do a head-to-head listening comparison between players that deviate enough from the typical output level of 2.0 volts to make one seem better than the other for that reason alone. If Player A has a maximum output level, say, of 1.88 volts and Player B’s is 2.12 volts — a 1-dB difference — Player B will almost always be preferred in a listening comparison regardless of any other measurable differences. Since you can compensate for below-par maximum output by turning up the volume setting slightly, it’s not a significant “defect” unless the output level is very low, because raising the volume will also increase the background noise.
Frequency response was of supreme importance in the heyday of the LP record. It still is important, but since so few digital disc players we’ve encountered deviate from audible — if not measurable — perfection in this respect, it is almost a moot point now. Deviations of 0.1 dB or less from 20 Hz to 20 kHz in any audio frequency-response measurement can be considered audibly perfect. Larger deviations may occur at both ends of the frequency range, especially with cheap or portable players, but unless they are very large (greater than 1.5 dB), it’s unlikely they’ll be audible either.
For all sampling rates above 44.1 kHz, the ideal upper-frequency limits for DVD-Audio performance are precisely half the sampling rate, as shown in our sample lab box. None of the DVD-Audio players we’ve tested so far get up this high — in fact, all of them cut off at virtually the same frequency and with the same decibel variation. That’s because many of them are using the same digital-to-analog (D/A) converter chips! Not to worry — if you’re old enough to read and understand this article, you can’t hear anything above 24 kHz anyway, and probably not even above 18 or 19 kHz.
Measurements of audio distortion (THD+N, or total harmonic distortion plus noise) are always stated in percent mainly out of tradition. I have yet to see a DVD player that has an audibly significant distortion percentage — greater than 0.1% if you’re really critical, or 0.3% if you’re not — certainly not at full output level (0 dBFS) and not even at our reference level (–20 dBFS). If you’re going to use a player’s analog outputs, anything more than 0.3% distortion should disqualify it from consideration, regardless of any other virtues it may have. But if you’re using its digital outputs, the only relevant distortion measurement is of the digital receiver or preamp you connect it to.
To be more useful in distinguishing one player from another, distortion should be measured like noise level, in decibels below some reference, and then you could compare the two figures (ideally, they’d be within 3 dB of each other). But that is a really radical concept to some, who take (false) comfort in the seemingly more friendly percentage concept. The problem is, perfect distortion performance is not zero but is limited by the resolution of the test signal.
For CD players, theoretically ideal distortion performance should be 0.00153% at 0 dBFS and ten times worse at –20 dBFS (0.0153%). For DVD-Audio players, the test disc we’re now using, which is the only one I’ve been able to track down so far, doesn’t have suitable signals for distortion measurements (but I hope this will be remedied within the year). In any case, lower distortion is better.
Defect tracking is the most variable measurement we publish since merely reloading the Pierre Verany test disc (which has data-layer errors in ascending calibrated sizes engraved on it) can produce slightly different results each time. Nonetheless, no better “objective” test for this characteristic is available. The minimum defect-tracking required of a player by the CD standard is 200 micrometers (µm). Ideally, a player should track defects up to the 3,000-µm maximum on the Pierre Verany test disc — in any case, the higher the figure in our lab results, the better.
Noise-Related Audio Performance
Four of our standard audio measurements are so closely related I’ll discuss
them together. Our noise-level, excess-noise, noise-modulation, and linearity-error
tests combine to make things difficult for a player. It’s hard to perform
well on all four tests simultaneously, and if a player does poorly in one of
these tests, at least one of the others will usually show nonideal behavior
as well.
The most conventional and easiest to understand is noise level, which measures how far the player’s inherent background noise (consisting of hiss as well as hum and buzzes from the power supply) falls below our standard reference level, which is not full output but 20 dB below that (–20 dBFS). All of our noise-level measurements use A-weighting, which (approximately) corrects for how the ear’s sensitivity to low-level sounds varies with frequency. We also use a special “dither” signal (random extremely low-level noise) for this test, not an all-zeroes data stream as in other methods of measuring noise, so the figure representing theoretically perfect performance is not minus infinity (–•) but is instead set by the resolution of the medium.
The resulting noise-level figure should be as close as possible to the theoretical minimum without going lower. For 16-bit CD audio, perfect performance is –75.9 dB. For DVD-Audio using 24-bit data, perfect performance would be 48.16 dB lower, or –124.1 dB. I don’t expect any DVD-Audio player ever to measure that low, however. Most players will be closer to 20-bit performance, or -100.0 dB, at which level a perfect DVD-Audio recording should have absolutely no audible background hiss at any reasonable listening volume.
Excess-noise figures explicitly measure a player’s departure from an ideal level of background noise. For a CD player, this test — which so far is exclusive to Sound & Vision — quantifies how much higher (or lower) than perfect the player’s output level is with four types of signal: 16-bit (EN16) and quasi-20-bit (EN20) dither, both with and without a low-level sine-wave signal superimposed, in a frequency band that’s squarely within the ear’s region of greatest sensitivity. Ideally, all excess-noise figures should be 0 dB, meaning that the player adds no noise or distortion to the dither signal in the frequency band we look at. In practice, we’ve seen 0-dB EN16 and single-digit EN20 performance with CDs played on the latest DVD-Audio gear. If the results aren’t exactly zero, then the “with sine tone” figures should be within 1 dB of the “without sine tone” figures.
When it comes to DVD-Audio playback, the excess-noise figure represents, again, the amount of noise added to a theoretically perfect 24-bit signal. I don’t expect ever to see 0-dB EN24 results. Probably the best that can be expected is approximately 20-bit performance, or excess noise around +24.1 dB. As long as a player comes close to that figure, its inherent noise is unlikely to ever be audible.
Noise modulation measures how much the background-noise and distortion levels
change as the signal level changes over a very wide range. Ideally, it should
be 0 dB. Typical, and excellent, values for CD playback are less than 0.5 dB,
with noise modulation figures as great as 3 dB usually being of no audible consequence.
Linearity error measures the level difference, plus or minus, between a –90-dB
signal on our test CD-R and the player’s analog output as it plays that
track. Such very low-level recorded signals often prove difficult to reproduce
accurately, but errors are still usually much less than a decibel (zero error
is ideal). A relatively large error (greater than 1 dB) often shows up also
as noise modulation (which can be generated by the same mechanisms as a negative
linearity error) or higher-than-theoretical noise or excess-noise figures (a
positive error).
Ranking the Readings
Given the generally high level of performance we’ve been seeing from recent
DVD players, not all the measurements we’ve been making are now of equal
importance for your buying decision. You can usually skim over the audio frequency
response and distortion measurements. All of these are unlikely to be less than
very good. The notes at the end of “in the lab” will mention anything
unusual.
Some other measurements might be irrelevant for a different reason, not because every player is equally good but because they don’t apply to your viewing situation. If your TV doesn’t have component-video connections, the two component-output measurements have no bearing. In-player letterboxing is very important if you’re watching on a 4:3 screen, irrelevant with a widescreen 16:9 set.
If I had to pare down the lab-test figures to the minimum that could be decisive in choosing one player over another, I’d rely on luminance frequency response and in-player letterboxing for video. On the audio side, the excess noise and noise modulation results should hold sway because, of all our audio tests, they’re the hardest things to get right.
Of course, such a minimalist view assumes that your mind hasn’t already been made up by some other, more subjective aspects of the players’ performance — things we can’t measure in the lab but that our reviewers always discuss in detail in their reports. How easy it is to use the remote control, for instance, or to find what you need in the onscreen menus are things that should weigh just as heavily in your purchase decision as the most important lab results, and maybe even more so.