Blind Testing is the Key Determinant of Audibility
Tom Martin -- Tue, 05/27/2008 - 07:23
digital (Andrew D) wrote:
"Ok boys, I've conducted properly organized, well thought-out, blind listening evaluations and always come to the same conclusion: cables don't make an audible difference in an audio reproduction system.
Those of you on this forum who are determined to stick to your "yes they do" guns: all I can suggest is that you submit yourself to a blind evaluation - one where there is [absolutely no way] for you to see which cables are in place at any time. In fact, the best case scenario involves listeners / loudspeakers in one room and reproduction kit / cables in an adjacent room separated by a closed door - a situation such as this:
http://www.canuckaudiomart.com/node/
...which I conducted for a group of enthusiasts on a Canadian Audio website. Let me go on record in stating that I am more than ready to retest [anyone] who offers to come forward with whatever cables they wish to show up with at my doorstep - any time.
There is a humorous follow-up to that evaluation here:
http://cdnav.com/cdnav/viewtopic.php?t=164 (read it to the bottom to get the gist)
In closing: Something like this:
http://www.thesourcecc.com/estore/Product.aspx?language=en-CA&catalog=On...
...works perfectly for RCA interconnects and something like this:
http://www.thesourcecc.com/estore/Product.aspx?language=en-CA&catalog=On...
...does the trick every time for loudspeaker cables.
Andrew D.
www.cdnav.com."
You are right, there is no difference between a $4000/meter interconnect and a couple of pieces of straightened-out coat-hanger wire - other than the looks (and of course the fact that coat-hangers are not shielded). For audio, wire is a conductor and conductors are conductors. While cables have a great affect on some signals at some frequencies, audio frequencies aren't among them. DC to 50KHz has no properties where inductive or capacitive reactance or even resistance (unless the cable was defective) could have the slightest effect on an audio signal over less than 10 or 20 ft. Then there MIGHT be a slight attenuation above 15 Khz.
The next time you find yourself contemplating the purchase of some pretty new cables with the price-tag of a tin of Iranian caviar, ask yourself this question: Have the cable "breakthroughs" advertised by this company been published or subjected to any form of scientific peer review? Has this company's findings with regard to cable effects over the audio range changed the way in which, for instance, the aerospace industry, builds its cabling? Have textbooks been updated to reflect any of the "research" done by such companies as Monster, Cardas or Nordost? If you find (as I'm sure you will) that the answer to all of these questions is NO, then I suggest that you take your money and buy that tin of Iranian caviar - or better yet, spend it on some music.
Without a doubt IN YOUR SYSTEM cables don't make a difference.
In my system they do.
End of discussion... :roll:
Steven Stone
Contributor to The Absolute Sound, EnjoytheMusic.com, Vintage Guitar Magazine, and other fine publications
Well, Steven, perhaps you think so, but I doubt seriously that even you could tell one cable from another in a double-blind or ABX test. Someone on this forum said that price and looks contributes a lot to our "perceived" value of audio products and I think that's true. A pair of $4000/pair interconnects just HAS to sound better than a pair of molded Radio Shack cheapies and so, they do.
Also, I don't believe that you have heard my system so I doubt whether you could know that I couldn't hear the difference between cables in my system (assuming such differences exist). I used to think that I could hear cable differences and I have some pretty high-end cables in my system. But using double-blind ABX testing on a megabuck mostly state-of-the-art system - not my own, unfortunately (MG-20's, Mark Levinson #33's, AR SP17, Sony XA777ES {my contribution to the afternoon's fun} and various interconnects from Cardas, Audioquest, Kimber, etc.) 6 experienced audiophiles (many of which contributed their own expensive cables to the cause) could not identify which cable they were hearing. When they tried, they were guessing, and the results were a statistical wash.
It's really funny. When you remove the sense of sight from many of these evaluations a lot of so-called differences vanish into thin air.
George Graves
George I don't doubt that you don't hear a difference.
I'm also not saying that I could hear a difference in your test in the listed environment with the listed equipment.
What I'm saying is all the test proves is that you and other test subjects could not hear a difference.
The test does NOT prove anything besides that fact.
Why?
1. DIfferences may have been too subtle to be percieved by listeners to the system in its test environment - dipole speakers use the room for much of their sound, and reflected sound relies on the room's sonic signature.
2. Short A/B/X testing is not the only way to make value judgments. Long-term listening is an equally valid for making sonic evaluations. Many sonic evaluations (such as listener fatigue) can't be done in a short-term listening test.
3. A/B/X is only one test methodology. It is not THE ONLY test methodology.
As someone much smarter than I once wrote, (Daniel VonRecklinghousen) "If something sounds bad and tests bad, it is bad. If something sounds good and tests bad, you're testing for the wrong thing."
I submit that A/B/X testing is the wrong test for subtle sonic differences.
No A/B/X test will ever prove me wrong on that point.
Steven Stone
Contributor to The Absolute Sound, EnjoytheMusic.com, Vintage Guitar Magazine, and other fine publications
George Graves comments about cable audibility inadvertently supports the thesis of my editorial: that blind testing of audio products leads to the wrong conclusions. To conclude that coat-hanger wire is sonically identical to any high-end cable is patently absurd. For a more detailed analysis of why, see my AES paper on the subject at
http://www.avguide.com/news/2008/05/28/the-role-of-critical-listening-in-evaluating-audio-equipment-quality/
Steven wrote:
"As someone much smarter than I once wrote, (Daniel VonRecklinghousen) "If something sounds bad and tests bad, it is bad. If something sounds good and tests bad, you're testing for the wrong thing."
I submit that A/B/X testing is the wrong test for subtle sonic differences.
No A/B/X test will ever prove me wrong on that point."
Steven -- thanks for this. Here's a genuine question aimed at you but open to all: if component A sounded good to you, and then you later found in a blind test that you could not reliably tell it apart from component B, which, when sighted, you had not liked as much, do you think your view of component A would subsequently change? Why?
On your blind testing editorial, I feel that I must comment. While I am obviously an audio enthusiast (I don't like the connotation that some give to the term "audiophile"), my vocation is metrology (the science of measurement). Because of the former, I can appreciate the contribution that critical listening plays in audio, but due to my latter affiliation, I cannot dismiss the importance of measurements and blind testing. Speaker cables and interconnects have always been a popular whipping boy, when it comes to this area of discussion, and while I appreciate the value (and cost) of engineering and its impact on the ultimate selling price, in the end, it's all about R, C, and L (resistance, capacitance, and inductance). With that said, any given cable may sound great on your system but may sound lousy on mine, and vice versa, since the source impedance of my amplifier, cable length, the input impedance of my speakers (and any resultant phase shift, etc., etc.) are different. When all of these other variables are combined with the electrical characteristics of any cable, the end results (how they sound) can't help but vary. Therefore, I can measure a cable's electrical characteristics and it can represent itself very well, in those terms, and sound like crap in a particular system, due to the aforementioned systemic interaction. But, to some extent, the converse can also be true, mediocre (relative to parametric measurements) cables can sound great on a system even though their measured values are not the best. In the end, however, I believe that a combination of measurements and blind testing is best, though perhaps it should be emphasized more that a reviewer's opinion of how a speaker sounds is just that - an opinion - and not be taken for absolute gospel. People do hear differently (and, as I previously mentioned, the system components can also make a difference, sometimes huge, though I don't think many people realize or consider how much impact it may have on the outcome of a review), but measurements (assuming the use of calibrated instrumentation) should be repeatable time after time after time and thus, an unchanging, reliable constant during any component evaluation.
Keep the good stuff coming,
-Dave Martson
Quote:George Graves comments about cable audibility inadvertently supports the thesis of my editorial: that blind testing of audio products leads to the wrong conclusions. To conclude that coat-hanger wire is sonically identical to any high-end cable is patently absurd. For a more detailed analysis of why, see my AES paper on the subject at...
First of all, the coat-hanger bit was, admittedly, a bit of hyperbole and meant not to be taken 100% literally. But this question is more complicated than whether or not blind testing can detect differences in cable, Robert. It goes to the very heart of a matter of our understanding of the forces at play here. As the above poster re-iterates, conductor physics is among the best researched and best understood areas of electronics and consists of just four characteristics: DC resistance, capacitive reactance and inductive reactance and dielectric properties of the insulation - all of which have negligible effect at audio frequencies (and somewhat above). I should know, I spent my first three years out of college in the Lockheed cable lab researching cable characteristics for aerospace. There is simply nothing about an audio signal that any piece of decently conductive wire can't carry unaltered and unadulterated. In other words, I would have been surprised if the panel in the ABX test that I attended HAD found reliably identifiable difference in interconnects.
The problem with purely subjective testing, even the way we reviewers do it is that without a reliable way to test what we hear, there is no way to tell whether what we hear is the result of actual differences or merely the result of our minds reacting to other stimuli at a less than conscious level.
Don't get me wrong, I'm not dismissing subjective evaluation as a legitimate tool for accessing the quality of music reproducing equipment. after all, as a reviewer for "The Audiophile Voice" for the last 12 or so years, I use this methodology myself and was taught it personally by the undisputed father of us all, J. Gordon Holt (who I am proud to call a good friend). It's just that I can't see throwing out double-blind testing either. When a double-blind test shows no differences between two similar products, I must conclude that a difference which makes no difference is no difference at all and if differences do exist and are hidden by such human failings as listener fatigue, or other causes, then we are making much ado about nothing. In the case of cables, if they do sound different, then that says that there is something about conductor technology we don't know and while that's possible, its also unlikely. And in the absence of any quantitative substantiation, we are, at worst, stabbing in the dark, and at best spending an awful lot of money for some very subtle "tone controls".
George Graves
[/Here's a genuine question aimed at you but open to all: if component A sounded good to you, and then you later found in a blind test that you could not reliably tell it apart from component B, which, when sighted, you had not liked as much, do you think your view of component A would subsequently change? Why?]
Although I doubt it was completely intentional, I see this as a straw-man argument.
It assumes a situation that may never come to pass and puts the responder in a position that has little direct interpolation with reality.
While not as extreme as "Do you still beat your wife?" it is still impossible to answer without being slimed.
So, I won't answer it. Sorry. :roll:
Steven Stone
Contributor to The Absolute Sound, EnjoytheMusic.com, Vintage Guitar Magazine, and other fine publications
Quote:The problem with purely subjective testing, even the way we reviewers do it is that without a reliable way to test what we hear, there is no way to tell whether what we hear is the result of actual differences or merely the result of our minds reacting to other stimuli at a less than conscious level.
For the sake of argument, turn this around. Suppose you started with blind tests and could hear no “statistically significant” differences between Cable A and Cable B. Then you listened to Cable A and Cable B in your home for a period of days or weeks on a variety of music you are familiar with and on a system you know quite well, and you did hear consistent differences. Would that then mean that blind-listening tests were removing "conscious levels" of reaction? And if so, why would you want to disorient yourself in this way when judging the different qualities of anything?
Moreover, how is a blind test any less "subjective" than a non-blinded test? Aren't you still listening for differences, and judging on the basis of what you hear—or don't? And if that is case, why would you assume that your hearing is somehow improved or made more valid by sensory deprivation of any sort, unfamiliarity, time constraints, test anxieties, relatively rapid changes in stimulus, and agreement or disagreement with a bunch of equally handicapped folks—not to mention any preconceptions (pro or con) that you may bring to such an experiment or any peer pressures that may be involved, the amount of experience you've had listening for subtle differences in sound quality in blind listening tests, the health and quality of your hearing, and (assuming, of course, that you are literally blindfolding yourself or keeping your eyes shut during these blind listening tests) the huge role that vision plays in helping us to "organize" what we hear, etc.?
Quote:Here's a genuine question aimed at you but open to all: if component A sounded good to you, and then you later found in a blind test that you could not reliably tell it apart from component B, which, when sighted, you had not liked as much, do you think your view of component A would subsequently change? Why?
Steve, I think you're wrong. This is not a strawman argument (and I don't think that the OP meant it as one, either). On the contrary, it goes right to the heart of the matter. The answer says a lot about about where you stand and how you see the importance of your own informed opinions.
First of all, I think that anyone faced with this result we be taken aback by it. It wouldn't be anything that any reviewer would expect - certainly not having previously spent weeks or months with the component prior to the blind-test. I would expect that I could pick that particular component out anywhere. When I found that I couldn't, I would probably choose to blame my inability to tell one component from the other on the test set-up, unfamiliar ancillary components, unfamiliar speakers, room, listening position, stress, anything. If one believes Bob Harley's weighty tome on the subject, there are any number of things that mitigate against being able to discern one component from the other in a blind-test.
Would such an outcome change my mind? I don't think so. Would it matter to me? Not so much.
jvalin wrote: "... how is a blind test any less 'subjective' than a non-blinded test? Aren't you still listening for differences ...?". The purpose of a blind test is to try to eliminate the influence of certain factors such as price, the manufacturer's reputation, and others' opinions of the component in question on the listener's judgment. This aspect of blind testing certainly reduces subjectivity.
Mr. Valin also lists some plausible reasons why blind testing might adversely affect the listener's ability to discern difference between components, including sensory deprivation, time constraints, and relatively rapid changes in stimulus. As I pointed out in a separate post (15, Jul Tue 2008 12:32 am in the thread http://forums.avguide.com/viewtopic.php?t=3914), a useful response to these criticisms would be to design blind tests that do not suffer from such flaws. That certainly seems feasible for components such as cables and interconnects, yet several of the audio reviewers posting here appear to reject blind testing entirely. I believe that both blind testing and observational listening have important roles to play in the field of high-end audio.
John Mitchell
After reading the editorial on blind testing I concluded that it did not address the main issue at hand in any way whatsoever, and merely underlined that any conclusions, based on listening alone, are suspect.
If I have correctly interpreted the source of the basic question, the reason this issue comes up so often is the concern that, if a reviewer knows the manufacturer and cost of a piece of equipment under evaluation, it can impact (perhaps only subjectively) the review conclusions. I think that we all have read reviews, of extremely expensive equipment, where significant performance issues were raised, but the item was still given a positive final assessment. The impression left is that the item would have been trashed if the reviewer had not known what it was! I know that all magazines refute this contention, but they also seem unwilling to do anything to test for this.
Most of the material in the editorial is not relevant to the fundamental question regarding does a reviewer need to see and know details about a piece of equipment in order to properly evaluate the sound quality? None of the mass "blind" testing performed (and lambasted at length in the editorial) is of relevance when considering a reviewer in his normal listening situation – but just not aware of what he is listening to.
Let us consider the Swedish radio tests. The only valid question to ask is if Bart Locatani needed to know which track was the original, and which track was the processed one, in order to hear a difference. If he could detect a difference, without knowing which track was which (a situation I consider to be by far the most likely), then he essentially conducted the type of blind test that would be appropriate for TAS reviewers. All of the other arguments regarding blind testing are nothing more than a smoke-screen..
Most likely Bart Locatani, being a digital expert, thought that he heard a difference and performed a spectral frequency analysis of the two tracks to see if he could identify the reason – which apparently was an easy task. This emphasizes the importance of measurement in audio, in addition to merely listening.
I have come across a somewhat similar situation myself. I could not identify exactly what it was about the sound of the Chailly Mahler 4, on Decca 289 466 720-2, that bothered me until I performed a spectral frequency analysis. To my astonishment I found that there is a frequency suck-out of about 30 dB, centered at about 9.2 kHz throughout the disc. At least I now know what it is (probably due to an equipment failure at the recording site).
It is an unreliable approach to rely on "critical listening" alone in the evaluation of a component. When one reads "represents one of the greatest technological breakthroughs in high-performance audio that I have experienced in my audiophile lifetime", one would not expect this analogue interconnect to add about 1% harmonic distortion to the signal (Stereophile, Vol. 28 No 8)! On another occasion, when reviewing an amplifier which had adjustments supposedly to be able to reduce harmonic distortion, the reviewer’s preferred settings were actually at the position that maximized distortion. Note that these two examples do not imply that the reviewers do not hear well, it is just that their reference is inaccurate (being able to adjust for maximum distortion by ear is really quite impressive). These two examples jumped into my mind, however they merely indicate a general type of trend that I find in all "subjective" reviews – they do not have a solid point of reference. With! Stereo phile, we do get the measurement information that enables us to "calibrate" the reviewer. I wish that TAS would start making measurements available to readers, and not hide reviewer preferences behind spurious arguments regarding "blind" listening and measurements being of no interest.
Tony Knight
[accidently posted this earlier in the wrong thread and deleted it, placing it here, as originally intended]
I originally said:
[/Here's a genuine question aimed at you but open to all: if component A sounded good to you, and then you later found in a blind test that you could not reliably tell it apart from component B, which, when sighted, you had not liked as much, do you think your view of component A would subsequently change? Why?]
Steve replied: "Although I doubt it was completely intentional, I see this as a straw-man argument. "
Steve, it was never my intention to erect a straw man or trick anyone. Rather, I was offering what I consider to be an interesting thought experiment on a key question but I can understand that it makes people uncomfortable. Let me try offer an answer that does not involve the respondent inviting anyone to abuse, dismiss or slime my position.
So, were this to be my experience, I might respond in one of several ways, simplified here for easy expression:
A) No change in my view of components. Fault the test - or at least try to identify problems in the method that might reduce my ability to listen properly, on the basis that I love A, and if I had a chance to hear A and B under my terms (sighted), I would be sure to pick it as my favorite. With respect, this is the typical sighted reviewer response, albeit articulated with greater subtlety.
B) Change in my view of the components that causes me to convert to the 'objectivist' camp. Give up sighted testing for serious comparisons and rely on blind tests to determine what I might purchase (actually quite hard to do but this is just a thought experiment). I might also start berating everyone else who does not believe in DBT (ok, I am throwing in a stereotype, none of us would ever really do this
C) Still preferring A over B in my system, now consider myself informed on the relative degree of difference between components. Start to entertain the thought that my preferences might be influenced by sound and other factors. Use this insight carefully.
Clearly I believe that the best option is C though I am sure I could construct a rational (though slightly weaker) argument for holding the other positions. From there, I believe I might start to think about why the blind test did not reveal my stated preference, learn to hear what matters for me, and perhaps develop a little humility to go along with my curiosity about what makes a component sound pleasing to me. Regardless, I could still enjoy A in my system and be satisfied with my purchase. I also would think it sensible to calibrate my listening preferences every once in a while through such tests, both to keep me honest with myself and to help refine my own ability to discriminate. I might also begin to design blind listening test methods that do reflect my more typical listening process.
While I do not expect we will ever see blind test data reported as standard in audio reviews, I do hope that we could have a less heated discussion of the appropriateness of certain test methods (and this forum gives me great hope, thanks all!). When most reviewers adopt a single position on the value of blind tests, it's difficult to present a convincing case to most people of the seriousness of our pursuit (not that this is the primary criterion). There are many flavors of blind testing that should be explored and I believe Revel have shown the value of such tests in at least the product design stage. Rather than contort ourselves with philosophical speculation on reality, I'd love to see us openly explore the opportunities provided by the range of empirical methods that remove knowledge of the component's identity from our evaluations. There is room for observational listening over the long term, but it should not be the only method we use, especially if we have the shared goal of enabling ever better reproduction of the wonders of music.
Peace
Julian
Julian:
My compliments on a superb post. As a scientist and an audiophile, I am tormented by the failure of DBTs to pick up differences that seem clear as day with sighted listening. So I do what any self-respecting person would do: go into deep denial. :)
One additional comment
Quote:...and perhaps develop a little humility to go along with my curiosity...
Surely there is a law prohibiting the use of "humility" in reference to matters audiophilia. [wink]
Best regards,
Larry Borden
"Digital finishes what the transistor began" James Boyk
robert_harley6 wrote:
If I have correctly interpreted the source of the basic question, the reason this issue comes up so often is the concern that, if a reviewer knows the manufacturer and cost of a piece of equipment under evaluation, it can impact (perhaps only subjectively) the review conclusions. I think that we all have read reviews, of extremely expensive equipment, where significant performance issues were raised, but the item was still given a positive final assessment. The impression left is that the item would have been trashed if the reviewer had not known what it was! I know that all magazines refute this contention, but they also seem unwilling to do anything to test for this.
It would be most interesting to read a magazine where the reviewing was done blind (without knowledge of what the product under review is or the cost/etc), with references to 'X' substituted with the correct product name after the review.
If these blind reviews were made relative to 'the absolute sound' (as per the manifesto) of some real music source, we might see more consistent & valuable reviews (though according to this criteria I would expect to see vastly more low rated reviews).
Also, it would be useful to have dB SPL ratings at which the reviews were made and some discussion of equal loudness as it relates to 'the absolute sound'.
Andy
www.SimpsonMicrophones.com - Next Generation Microphones
www.SimpsonMicrophonesArchives.com/Early_Music_A.mp3 - www.SimpsonMicrophonesArchives.com/Early_Music_B.mp3
Post new comment