Guest Blog: It's Time to Admit that We Don't Know Everything

Posted by: Tom Martin at 4:04 pm, January 21st, 2010

It is this fourth factor - the impact of cognition on the outcome - which makes DBT as applied to audio quite different from DBT as applied to, say, drug testing. It is often observed in audio DBT, in cases where subtle differences are uncontroversially acknowledged to be present (such as tests of lossy audio codecs), that "experienced" listeners are more statistically likely to identify differences. Why should that be? It's possible that experienced listeners are experienced because they can hear better, but given the disproportionately high representation of older males in the audiophile community, that seems unlikely. A more likely explanation is that experienced listeners just know what to listen for. When you think about it, this is not surprising. If we do a DBT on two different Stradivarius violins, who is more likely to reliably distinguish them? An untrained listener off the street, or the person who has owned and played one of the two Strads for the last 20 years? Any listening test - blind or not – conflates hearing and cognition, and is subject to deficiencies in either of those.

Bob Stuart of Meridian Audio, digital audio pioneer, degreed in both audio engineering and psychoacoustics, points out one obvious problem with DBT as applied to music listening: the “problem” of memory. If component B raises a subtle aspect of the music to the conscious level (the actual lyrics to Louie Louie?), you are now very likely to hear it when switching back to component A as well – because you remember it. People aren’t test instruments – they can’t reliably be reset to the same initial conditions. What you hear when you listen to music is a function both of your low-level sensory apparatus, and your experience. There is no way to construct a music listening test, DBT or otherwise, which conclusively proves what differences can and cannot be heard, because hearing can't be isolated as an independent variable.

A further observation about the limits of audio DBTs falls out of this discussion. In a double blind test of, say, drugs, we not only assume that the test subjects’ cognition (outside of the placebo effect) does not impact the results, but we also ignore statistical "flyers". If one test subject gets spectacularly better while taking the test drug, but nobody else does, the drug has failed. We chalk up the flyer to some combination of factors not applicable to the general population. But here's the thing: audiophiles are "flyers" by definition. The differences between the two Stradivari are of no import to the general populace, but Itzhak Perlman is a different story. The ability to detect woodwind sound reflecting off the orchestral shell is of no import to the general population, but it is to me. Many audio DBTs are oriented more at what the “average” person in a test population can or can’t “hear”, than in ferreting out performance differences at the margins.
 
Yes, single blind listening is obviously subject to serious errors such as confirmation bias, and attempting to confirm results through other mechanisms such as measurements and double blind tests is something "subjectivist" magazines should do much more often. And the more extraordinary the claims for a component are, the more scrutiny they should receive. At the same time, I'll submit that many advances in audio reproduction were introduced by engineers who used their ears as their primary tool. I remember well the controversies surrounding digital audio in the early days, 30 years ago. Objectivist audio equipment reviewers, using the measurements they'd used to evaluate the analog world, found that digital audio measured essentially "perfect", and their ears confirmed what their measurements told them. Many professionals who cared mostly about sound, however, like recording engineers and subjective audio reviewers, insisted that these early digital recordings didn't sound "right" in important ways. And sure enough, over time, the causes of various digital audio artifacts became studied and understood, measurements were devised, listening tests which exaggerated the phenomena educated audio professionals on how to hear these things, and a whole host of technical improvements ranging from dither and noise shaping to jitter reduction to innovative anti-aliasing and reconstruction filter design, among many others, were employed to attack and reduce the unique problems of digital audio, to the point that it now comes very close to meeting its original promise. What is especially ironic is that some of those types who proclaimed digital "perfect" way back then (hello, Brad Meyer) now take the line that, of course, those early systems had some little issues but we've fixed all of them now! I reserve the right to be skeptical.
 

I close with a few lessons from my own experience as an objectivist audiophile: