The Foibles of Human Perception

Tom Martin -- Wed, 06/11/2008 - 16:48

Robert Harley posted in the D/A section of this forum an interesting item that fits with this section:

TAS reader David Sanford sent me this e-mail in response to my reply to his letter in TAS 182:

Thanks for publishing my letter in the June/July 2008 issue.
I don't really question the audibility of jitter. However, I have to be skeptical when someone tells me that timing variations between clock cycles of 7 picoseconds, (which is 7 billionths of a millisecond) are audible. I need a bit more scientific evidence than what one person claims he can hear.

I have been skeptical of people's perceptions ever since my high school physics lab. The teacher passed around two objects and we were to write down which one we thought was heavier. The first object was a small clear glass jar containing mercury. The second object was a larger clear glass jar containing some solid objects. Both jars had strings attached and each student was allowed to inspect the jars simultaneously any way he liked including holding them by the strings. Just over three-fourths of the class wrote down that the smaller jar containing the mercury was heavier. After we all gathered around and weighed the jars we found that the larger jar in fact weighed twice as much as the smaller jar!
Why were most of the students wrong, when the weights of the jars weren't even close? I don't know, but I realized that day that you should question the accuracy of human perceptions. In high end audio logic, however, many would argue that the fact that the larger jar actually measured heavier isn't important. What is important is that the students perceived the smaller jar to be heavier and therefore it must be.

Regards,
David Sanford
Back to top
View user's profile Send private message Send e-mail

discman

Joined: 23 Feb 2008
Posts: 104

PostPosted: 09, Jun Mon 2008 10:36 am Post subject: Reply with quote

If I understand Johnson's point, then Sanford (who may not have seen Johnson's message) misses it. Sanford, I believe, is thinking analog. But this is digital. In digital signals, time has meaning that is different than what it means to humans in the analog domain. Thus, we are not talking about direct human perception of picosecond time, but the digital meaning of picosecond time.

If for example, we had a time-based encoding system operating in the terahertz range, then we might assign different frequencies to different digitally encoded numbers. A difference of even 1 picosecond in such a system would change the digital representation of the signal. A 7 picosecond variation might completely change the signal (close to 100% distortion, perhaps?).

Now in the case of CD, the frequencies are much lower, and I don't know enough about the encode/decode schemes to know whether the data error introduced by 7 picosecond timing variations is material. The point is, there are two issues: 1) is there a data error introduced by jitter and 2) is the error perceivable. The former seems at least plausible, and until we know the type and magnitude of the error I don't see why we should assume it would be hard to hear.
Back to top
View user's profile Send private message

discman

Joined: 23 Feb 2008
Posts: 104

PostPosted: 09, Jun Mon 2008 10:40 am Post subject: Reply with quote

Sanford, btw, may make another interesting point (unintentionally): his student buddies were very accurate perceivers (of density) and bad explainers (their theory was out of alignment with their perceptual system). We should not confound these two.

Tom Martin -- Wed, 06/11/2008 - 16:50

Maybe the really interesting point that Sanford raises is our willingness to accept theory as a better arbiter of reality than human perception?

CEO and Editorial Director, Nextscreen LLC

Robert Harley -- Thu, 06/12/2008 - 09:07

The rejection of the role human perception plays in understanding reality is a fundamental belief of audio “objectivists” and the core of “The Great Debate.” (The Great Debate is the name given to the controversy between audio “objectivists” and “subjectivists.”)

Audio academics tend to live in a theoretical world, rejecting anything that can’t be quantified and expressed through mathematical symbols. Of course, one’s feelings when listening to music is the antithesis of the hard data “objectivists” demand. In fact, the leading audio “objectivists” refuse to listen for themselves. I address this issue in my Audio Engineering Society paper “The Role of Critical Listening in Evaluating Audio Equipment Quality” and quote the leading objectivist stating why he rejects personal experience in evaluating audio equipment. You can read the entire paper here:

http://www.avguide.com/news/2008/05/28/the-role-of-critical-listening-in-evaluating-audio-equipment-quality/

There’s an entire book devoted to the question of human perception in knowing reality: Personal Knowledge: Towards a Post-Critical Philosophy by Michael Polanyi. Originally published by the University of Chicago Press in 1958, it was reprinted by Harper Torchbooks in 1964. It’s quite long and dense, but endlessly rewarding. One could interpret Personal Knowledge as the philosophical foundation of the observational approach to audio reviewing.

Tom Martin -- Thu, 06/12/2008 - 09:29

Even the word choice is interesting here: objectivists vs subjectivists. If we take into account that this is really a three way argument:

quantified/theoretical objectivists

observational objectivists

and

subjectivists,

then what I get from Sanford is that we are often more willing to believe a mediated (e.g. spectrum analysis or distortion measurement or theory of time audibility) view of perception that we are a human view. There are, of course, reasons for this. But when it is called out so clearly it also seems a bit weird.

I should add that one needed choose just one method.

Tom Martin -- Thu, 06/12/2008 - 09:31

Maybe TAS should do an interview/panel with Robert Parker (the wine critic to end all wine critics) and other observational objectivists to discuss method and call attention to the rigor that goes into this approach. Michael Reichman from Luminous Landscape (or Sean Reid from ReidReviews) would be interesting observational objectivists from the photography realm.

Robert Harley -- Thu, 06/12/2008 - 20:08

That's an interesting idea. I wonder if there's debate in other fields of connoisseurship.

Robert Harley -- Thu, 06/12/2008 - 20:10

There's actually another way of looking at the world in which there's no distinction between subject and object. It's called the Metaphysics of Quality. Check out moq.org.

Tom Martin -- Fri, 06/13/2008 - 08:58

MOQ is interesting, though my reading would be that it attempts to get around the subject-object division. Here are two quotes from a paper by Anthony McWatt on the Metaphysics of Quality:

"The Metaphysics of Quality subscribes to what is called empiricism. It claims that all legitimate human knowledge arises from the senses or by thinking what the senses provide. Most empiricists deny the validity of any knowledge gained through imagination, authority, tradition, or purely theoretical reasoning. They regard fields such as art, morality, religion, and metaphysics as unverifiable. The Metaphysics of Quality varies from this by saying that the values of art and morality and even religious mysticism are verifiable and that in the past have been excluded for metaphysical reasons, not empirical reasons. They have been excluded because of the metaphysical assumption that all the universe is composed of subjects and objects and anything that can't be classified as a subject or an object isn't real. There is no empirical evidence for this assumption at all. It is just an assumption."

So why introduce a new metaphysics?

This is probably explained best by my Norwegian friend Bodvar (Skutvik) and the quality/quantity paradox in subjective-objective thinking (see diagram on subjective-objective (or "SOM") metaphysics):

"In the objective world there are no qualities, only quantities: sight-colours are various wavelengths of the electromagnetic spectrum; sound-music are air pressure waves; smell-odours are molecular configurations, as is taste and touches are pressure sensation. Nowhere out there is quality (or values) to be found. The impacts on our sense organs are transmitted into electrochemical impulses travelling to the brain where it is translated back into our subjective perception. There is NO direct connection between the two realms ...if you start with the subjective/objective metaphysics (or the mind/matter idea if that sounds less "metaphysical")... subjectiveness is subjectiveness from here to eternity as is objectiveness; nowhere does the two overlap."

As can be seen from the diagram on SOM, quality is on one side of a metaphysical chasm and quantity is on the other. Quantity is perceived as inhering in substance, qualities are perceived as being non-substances. They are mutually exclusive and should therefore not be able to have an effect on each other. However, the fact that your mind can decide to move your little finger (a physical object) and a few pints of beer (a physical substance) can alter your mind totally dispels this idea. There is a serious metaphysical problem here.

The full McWatt paper is here:

http://www.moq.org/forum/mcwatt/anthony.html

Tom Martin -- Tue, 06/17/2008 - 07:34

I think that for many practical people, unconcerned with philosophical rigor, that the blurring of the subject-object distinction is the source of much discomfort about audio testing. What I hear on this forum is:

-- If information about what a speaker or amp is doing is judged by a listener and then that listener reports on what he heard, the subject is somehow involved

-- One possibility is that everything being reported is subjective; that is, the listener's (reviewer's) reaction is specific to that listener and thus relatively meaningless to me (another person)

-- If that is the case, then I want instrumented measurements. This removes the subject (mostly) from the equation.

-- But if I think about instrumented measurements, I have removed subjective foibles from the evaluation at the expense of meaning. In other words, I now have pretty reliable instrumented data, but that too is very hard to relate to what I will actually hear.

-- So, another possibility is that listeners (reviewers) can reliably describe what they hear (observational objectivity), and if I listened, I would hear the same things.

-- That would be the most satisfying case, but the press should endeavor to show that they have methods and people that make observational objectivity happen

-- And, then, I'd like the reviewer's value judgement of all this. That is because I want to know if differences "are worth it". Yes, I know that is really subjective, but so what? I want it.

Robert Harley -- Tue, 06/17/2008 - 10:18

The basis for my belief in the supremacy of observational listening over measured data is not only philosophical, but practical. I’ve listened to, and measured the technical performance of, perhaps more than a hundred CD players and D/A converters. The measurement regimen included low-level linearity, spectral analysis of the noise, distortion, intermodulation products via FFT analysis, jitter, noise modulation, output impedance, channel separation, power-supply isolation, etc.

I found no correlation between any specific sonic characteristics of the player and the measured data. Some products that performed well on all these tests were musically uninvolving, while others that measured poorly were extremely engaging.

If the sound is different, then signals are different. But trying to assess the musical performance of a CD player with these tools is like trying to measure one one-thousandths of an inch with a yardstick. Perhaps a better analogy is that we’re trying to measure one one-thousandths of an inch with a vacuum meter. Some sophisticated measurements, such as those developed by Keith Johnson, can give some indication of specific distortions that can be correlated to specific sonic characteristics, but the idea that we can describe how a product sounds by looking a measured data is absurd.

If I had to choose a CD player to live with for all my music listening based on ten hours in the test lab or ten minutes in the listening room, I’d choose ten minutes in the listening room.

Tom Martin -- Tue, 06/17/2008 - 10:45

I fully agree. Still "the press should endeavor to show that they have methods and people that make observational objectivity happen".

Julian -- Wed, 07/09/2008 - 20:07

First off, thanks to TAS and Robert in particular for being willing to have this debate -- let's hope we can keep is civil and avoid stereotyping people as extremists. For some of us, the issue of blind testing is not about dismissing differences or thinking audiophiles are dupes for buying expensive gear. I am an audiophile and a believer in the value of blind testing - one can be both. And I happen to own a lot of expensive gear too.

I do however, have to flag a couple of points you make Robert and suggest there might be some conflation of terms and positions which we would do well to clarify. I offer the following in the spirit of constructive discussion. You note:

"The rejection of the role human perception plays in understanding reality is a fundamental belief of audio “objectivists” and the core of “The Great Debate.”

I am not of this view. Believers in blind trials do not reject human perception at all, instead they recognize that human perception is complex and influenced by multiple variables. To claim A 'sounds different' than B when one can see A and B is to allow variables other than sound to influence the judgement. If anything, this is a recognition, not a rejection, of human perception and its complexity. Of course I realize some hardline folks think visual cues must automatically trump auditory cues to the point that sonic interpretations are completely overwritten by visual perception, but that is an extreme argument, not one that most people who value BT hold.

further you say when noting the limits of measures in the test lab:

"I found no correlation between any specific sonic characteristics of the player and the measured data. Some products that performed well on all these tests were musically uninvolving, while others that measured poorly were extremely engaging. If the sound is different, then signals are different. But trying to assess the musical performance of a CD player with these tools is like trying to measure one one-thousandths of an inch with a yardstick. "

Yes, this is so true. But does this not point first to the limits of our measuring devices rather than the suggest 'objectivity' is flawed? Many Blind Test folks are not interested in looking at measurements at all, other than the measurement of how many people could distinguish A from B when listening only, not when listening and looking. The lack of good test lab measures is not the point here, other than to wonder why magazines keep pushing some measures when we all know these have no real correlation with the sonic experience that will result (but I confess, I am not aware of anyone performing a serious meta analysis or regression analysis of such data, maybe there is a hidden correlation in that set somewhere, but I doubt it).

My experience is that there are real sonic differences between components but these are often not huge, but sometimes they are. If one is willing to pay a fortune for a small difference, that is a personal choice and not pertinent here, each to their own I say. Sure there are limits to blind testing but that is true of all methods (and blind testing is a method, it is not a measure). I am not wed to any specific test method, but I do believe that one cannot dismiss the value of sightless listening by appealing to the limits of lab measures based on engineering qualities or by noting that there are flaws in other people's published blind tests.

What is missing here, of course, is an examination of the variability in listeners, and it is here that I think people get most annoyed. That experienced listeners can hear differences that others cannot should not be underestimated. The connoisseurship aspect is vital and real. After all, we have to train doctors to read X-rays, and each moves from seeing a blurry mass of grayscale to later being able to distinguish tumor from cloud -- and it ain't the X-ray that's changed, it's their perceptual skills. It is no surprise to me that the TAS and S'phile reviewers can hear these differences, but it is a real disappointment to me that so many of you disparage blind tests.

If anything, blind tests can give us perspective, they need not be invoked as an invalidation of high-end audio, which happens too often. As I said, I live with both, and I fully appreciate the costs, limits, and complications of performing reliable blind tests. I also love my system. Let's learn together.

Peace

Robert Harley -- Thu, 07/10/2008 - 09:15

Thank you for your thoughtful approach to the issue of blind testing. Unfortunately, blind testing has long served as a bludgeon by which audio "objectivists" seek to discredit "observational" audio. It has become a wedge issue, leaving little possiblity of finding common ground in the quest to better understand the relationships between differences in signals and human musical perception. Your post is one of the most reasoned and least-polarizing statements on the subject I've read.

John Mitchell -- Tue, 07/15/2008 - 17:24

Robert, you wrote "If the sound is different, then signals are different." This statement seems to reveal an apriori assumption that psychological factors such as our expectations, price, manufacturers' reputations, and the opinions of others do not affect the listening experience. But it is well known that such psychological factors do affect human perceptions, and it is for precisely this reason that blind testing, properly designed and implemented, could serve to validate (or invalidate, as the case may be) the results of objective listening, as I wrote in another thread (http://forums.avguide.com/viewtopic.php?t=3914).

A similar sentiment was expressed by Julian in his post above (09, Jul Wed 2008 9:07 pm): "To claim A 'sounds different' than B when one can see A and B is to allow variables other than sound to influence the judgement. If anything, this is a recognition, not a rejection, of human perception and its complexity."

The suggestion that even expert audio reviewers might be unconsciously influenced by psychological factors is not an attack on audiophiles. It merely recognizes that audio reviewers are human. I'm sure many readers of The Absolute Sound and other audio magazines would be very interested in an initiative to use blind testing in conjunction with observational listening. Cases in which blind testing confirms the conclusions of observational listening would bolster the reviewer's credibility, and cases in which it invalidates those conclusions might save readers a lot of money.

John Mitchell

jfm -- Thu, 07/17/2008 - 05:27

It would be interesting to read about other fields that involve the senses and whether measurements validate human perceptions.

Two examples: wines and perfumes.

Are there measurements of taste and smell that experts and aficionados in those fields rely on? Or do human perceptions provide the benchmarks?

It seems that people in those fields are hardly as argumentative about this subject than audio folks?

Robert Harley -- Thu, 07/17/2008 - 08:39

Interesting that you bring up perfumes. Many consumer products are scented, from food to soap to toothpaste. Most of the scents are provided by just a few companies you've never heard of, yet produce a substance we come in contact with every day. The development of specific scents is an exacting chemical process, but one that is guided by highly skilled individuals who rely on nothing more than their noses. These individuals are among the highest paid employees of the company, and are famous within the small circle of professional smellers. It was reported that one such person smelled a new scent developed by anther company (he wasn't told which company) and he correctly identified the specific individual who created the scent purely on the basis of knowing the other individual's aesthetic. The companies who employ these people take out special life-insurance policies on them because their skills are so valuable.

These people know little or nothing about the chemical composition of the scents. Conversely, the chemists are clueless in determining whether a specific scent is good or not.

This isn't a case of measurement (or chemical analysis) validating human experience; in this field, human sensory skill and connoisseurship are king.

Lear -- Thu, 07/17/2008 - 09:18

Same deal with wine. Robert Parker is a key figure. Purely observational in approach. I know of no chemical analysis that is used by reviewers or consumers.

In wine, however, there is much debate about whether Parker's (and other's) results are reliable. On top of that, there is debate whether the issues about Parker's reliability are being raised by people whose economic livelihood is threatened by Parker's results.

It seems to me that with wine and perfume in mind, some of the debate in audio is misplaced. The quantitative school seems to assume that "unconcious" factors skew the result. While that could be, there isn't a lot of evidence that I see (hence the shouting matches?) to that effect. But, unlike wine and perfume, audio involves judging components within a large and quite complex system. That is the chief challenge, and it isn't obviously made easier by quantification. That's because a related issue is that the objective function for quantification isn't well described or agreed upon.

John Mitchell -- Thu, 07/17/2008 - 10:50

Lear wrote (17, Jul Thu 2008 10:18 am): "The quantitative school seems to assume that 'unconcious' factors skew the result. While that could be, there isn't a lot of evidence that I see (hence the shouting matches?) to that effect."

The purpose of blind testing is to try to eliminate the influence of such unconscious factors on the listener's judgment, and as long as audio reviewers refuse to use blind testing as part of the review process, there will continue to be a convenient lack of evidence regarding the influence of unconscious factors.

By the way, I did once read of a blind taste test of wines. The rankings did not match the rankings by price or reputation.

Of course, blind testing is different from using quantitative measurements to assess quality. It seems clear that a combination of all three approaches (observational listening, blind testing, and quantitative measurements) is most useful, yet we have on the one hand the "quantitative school" refusing to recognize the value of observational listening, and on the other hand the "audiophile school" refusing to recognize the value of blind testing (and perhaps quantitative measurement).

Tom Martin -- Fri, 07/18/2008 - 07:53

I think John makes a useful point, which is that there is probably value in blind testing, observational listening and quantified measuring.

I think we should be clear about how we got here, though. The "audiophile school" isn't saying that blind testing and quantified measuring are not useful. Rather, the assertion has been made that human perception is unreliable, and therefore that blind testing is obviously better; that assertion is being challenged.

If you read John's post carefully, you will see the presumption of blind testing's superiority. One cannot prove that unconscious factors are present simply by showing that blind testing produces different results than unblind testing. That would only show that the methods are different (though if you assume blind testing is better, then the presence of difference 'proves' unconcious bias). The presence of unconscious factors requires the addition of a theory of the mind and tests of that theory and even then would only show correlation.

I believe some testing of this has been done (the one I remember was of wine!) and I think the researchers found that knowledge of price of the wines mattered. If I remember correctly, what they "found" was that price actually increased the pleasure of drinking the wine. Now what do we do? Do we say, 'well, then some of the cheaper wines were really better'? Or, do we say 'well, real consumers of wine will know the price, so the unblind model is a better reflection of reality'? Or do we say 'so what, because such a test was not a test of wine critics who may be able to focus more on the consistent attributes of the wine'? Or do we say 'so what, because blind testing of wine tasting seems a bit more appropriate because the test timing mirrors the timing of the actual experience much more closely than it does with music (10 sec of music isn't experientially at all like 60 minutes)'? Or do we say 'the perceptual change due to being under the gun to accurately taste wine is at least as big a perceptual bias as knowledge of the price of wine'? Or do we say 'blind testing has the problem of requiring stimuli that reveal differences which is a priori hard/impossible when the purpose of the test is to discover those differences'?

I think that leads us back to John's key point: no single method is best because each method has issues/risks. Quantitative measurements will be extremely hard for consumers to use (they are hard for engineers to use!). But blind testing has its place, which seems to me to be primarily be some measure of whether there are gross differences between products (an interesting and useful bit of knowledge). Observational listening also has its place, and it seems useful for discovering subtleties of real musical value. I don't agree with Julian above that these are Golden Ears Only subtleties. I do 3-4 reviews a year, and I'm almost always amazed at how obvious the differences between components are. I can't imagine that someone who focused on it wouldn't hear those differences. They might not care, but they could hear what I hear.

jfm -- Fri, 07/18/2008 - 17:11

The differences between audio components can be difficult to discern in the short term, but they become all too apparent in the long term. That would be one of the limitations of blind testing.

John Mitchell -- Fri, 07/18/2008 - 17:22

tmartin: Thank you for your thoughtful reply.

You wrote: "If you read John's post carefully, you will see the presumption of blind testing's superiority." I'm not sure what gave you that impression. I wrote that "a combination of all three approaches ... is most useful", and in a related thread, I suggested using blind testing in conjunction with observational listening (Blind Listening Tests are Flawed: An Editorial, 15, Jul Tue 2008 12:32 am). I never suggested using blind testing alone, nor did I deny the usefulness of observational listening.

I believe that the contributors to this thread agree that the perceived sound of an audio system (as a conscious phenomenon) is a function of two things: the physical sound (i.e., the vibrations of the air, etc.) and various psychological factors. If a reviewer describes the perceived sound of a system based only on observational listening, the reader has no way of knowing what aspects of the perceived sound were due to the two types of influences (physical and psychological). I am suggesting that using blind testing as an adjunct to observational listening can help determine which, if any, aspects of the perceived sound were due solely to psychological factors. This approach would provide more information for readers to use in making purchase decisions.

Note that even if someone knows that product A sounds better than product B only when he (or she) is aware of which product he's listening to, he might nevertheless choose to purchase product A for the simple reason that A does in fact sound better to him, regardless of the reasons why!

The main argument that's been offered against blind testing seems to be that it does not and cannot eliminate psychological factors, but that it merely eliminates certain such factors while introducing other psychological factors (e.g., stress due to being "under the gun", disorientation due to being blindfolded), and so it cannot help "objectivize" the results of observational listening by determining which aspects of the perceived sound are due solely to the physical sound. I take exception to the "cannot". It's clear that, at least for some audio components, "blind" testing need not involve blindfolding, tests of short duration, or rapid switching between music samples. The essential point of "blind" testing is that the listener should not know what component he's listening to. I believe that if reviewers such as Robert Harley put their minds to it, they could design blind tests that do not suffer from the flaws that have been mentioned (again, at least for certain audio components; I doubt that this would be possible for loudspeakers, for example).

Finally, in discussing a blind test of wines, you wrote: "... price actually increased the pleasure of drinking the wine. Now what do we do?" My response is that having this extra information is beneficial to wine drinkers, and so wine reviews could be made more informative by incorporating blind testing as an adjunct to "observational drinking" (not as a superior replacement for it). Each individual can decide how to use this extra information as he or she sees fit.

You expressed other interesting ideas, but I don't have time to respond to them all. Once again, thanks for taking the time to reply.

Tom Martin -- Sat, 07/19/2008 - 22:32

John -- good stuff, thanks.

I see the presumption of blind testing's superiority in you prior post because of my understanding of the logic in your critique of observational listening ('as long as audio reviewers continue to...') based on the lack of blind testing and by noting that blind tests produce different results. When a methodology is said to be problematic because another methodology isn't used to substantiate it, the latter methodology must be (assumed to be) superior. Similarly, when differences are used to demonstrate the superiority of a method it must be because of an assumption of superiority of that method. Probably you didn't intend those implications, but they are there logically.

I think you are close to the heart of the matter when you point out that blind testing seems to be a way to get down to just the physical stuff without all that psychological stuff mucking things up. That seems nice at first blush, but I don't see that blind testing does that as conveniently as its proponents would have it. Blind testing is quite unlike what we really do when listening to music, so it seems to me to be automatically under the gun to demonstrate that it isn't radically flawed. And, yet, it seems to me that blind testing is 'given a pass' by many consumers and assumed to largely eliminate 'obvious' flaws in observational listening. Where is the explanation of how something very unlike real experience can be made meaningful to real experience?

Of course, observational listening also has to address two problems related to this. One is that observational listening by reviewers has to use words or pictures or numbers to relate its findings to others. This is a big issue, but one that any review-based methodology seems to suffer so it seems to me not to be a particular disadvantage of the observational methodology. The other is that observational listening clearly needs a way to get around the bias due to knowing the brand and price of equipment under test.

The approach we have used to the brand/price bias issue seems good but not great: hire professionals, have them do lots of listening to lots of equipment, have them attend live musical events, and then ask them to work very hard to convey what they find, and finally have others do the same and then judge whether the work of their colleagues is good (or request changes).

I'm sure this doesn't work perfectly, but I have to say I don't think the imperfections are mainly due to brand/price bias. That's because, as one of the participants in this system, I have heard a lot of the equipment that others have reviewed and I hear what others describe pretty easily and regularly. Maybe I have the same brand/price biases, but then it is weird that I agree with my colleagues descriptions, but not often with their thinking of how they'd spend their money.

Much of what is being described is from one point of view pretty subtle, and I think value judgements about those subtleties are the hardest part (actually, I think they are basically impossible to do for others). If we want blind testing or some other methodology to show us the value system we have, that seems unnecessary, though understandable. That is, if we believe that subtle differences don't matter, then we could insist on a methodology that is not geared to find them.

I think in most of our work at TAS and Playback, we have always assumed that of course we want to know about the subtleties as well as the big differences. If that is the case, we have further reasoned that while blind testing would reveal big differences, so will observational, objective listening (and thus why bother with blind testing which is difficult to do well -- much ado for nothing). But, and I think that is a very big but for some people, what could be desirable about blind testing is precisely that it seems to reveal only differences that almost anyone could hear. Thus it offers the reassurance that money spent on equipment described as having certain attributes will turn into a difference actually heard. That seems rational.

John Mitchell -- Mon, 07/21/2008 - 18:59

I don't consider blind testing "superior" to observational listening in the sense that blind testing should be used exclusively. I do believe that blind testing is superior for certain purposes, and as I've said, I think a combination of the two methods is generally superior to either one used alone.

You wrote: "Blind testing is quite unlike what we really do when listening to music ...." One of the points I've been making is that, at least for some components, blind testing doesn't have to be very different from what we really do when we listen to music. For example, issue 182 of TAS contains a review of Bybee Technologies Golden Goddess Speaker Bullets. These are small components that attach to a speaker's binding posts. For a blind test, an assistant could occasionally swap out speaker cables attached to the Bullets with the same type cables terminated without the Bullets, according to some convenient schedule. The terminations could be covered with a piece of cloth (for example) to prevent the reviewer from knowing which set of cables was being used. The review could take place over days or weeks, in the reviewer's home, with the reviewer listening to music in the usual way. What's so "radically flawed" about that approach?

One more comment: this may be old news to you, but I recently read an interesting account of what must be one of the first blind tests ever conducted for audio equipment, by one of the first audiophiles, Thomas Edison (in the book "From Tin Foil to Stereo: Evolution of the Phonograph"). Edison, who was very devoted to achieving realistic reproduction of the human voice, arranged a series of public "tone-tests" (circa 1916) wherein a singer would perform alongside a phonograph recording of himself or herself singing the same piece of music. According to the book, "a startling climax was provided when during the latter part of the program the stage would be darkened, ostensibly so that the auditors could guess as to when the artist was singing and when not. Suddenly the lights would come on, revealing that the singer was no longer on the stage. Invariably, each person in the audience would give an involuntary gasp, as he realized that he had been utterly unable to tell when the artist had actually ceased singing."

Tom Martin -- Tue, 07/22/2008 - 06:24

If you simply have blind-to-brand-and-item-under-test listening, off the top of my head, that would be a lot more like real music listening. Seems interesting.

My central point is that if we wish to say that multiple methodologies should be used, we also should state their individual advantages. We could say, for example, that we should use multiple utensils for eating, but if we say those are a fork, a hammer and a laser, it helps to state the use of each; I would say in this case such a statement would reveal that two of the three are not useful.

Eliminating brand/model bias seems laudable. I do wonder if there is, among some writers, the second motivation to using an additional methodology that I mentioned: the desire to separate the big differences from the hard to hear.

Robert Harley -- Tue, 07/22/2008 - 21:01

The quote from the Edison demonstrations is interesting. Every generation has believed that its audio technology achieved absolute realism. Another quote from that period is Metropolitan Opera singer Anna Case who said "Much to my astonishment, I was unable to tell the difference between my own voice, and Mr. Edison's recreation of it. (1919).

The same phenomenon was repeated in the 1960s with AR's demonstration of the AR-1 on a stage with a chamber group. Those in attendance reported that they could hear no difference.

John Mitchell -- Wed, 07/23/2008 - 17:07

Anna Case's quote, mentioned by Robert Harley, also appears in the book I mentioned, along with several other testimonials from that time claiming that no one could distinguish between the live singer and the reproduction (using Edison's phonograph and recordings). It makes one wish for a time machine.

The book includes another interesting anecdote about a skeptic who attended one of the "tone-test" recitals by Anna Case. This man (an "artist", presumably meaning a musician) insisted that he could detect a difference between the live and recorded voices, and to prove it, he sat on the floor so that he could not see the stage. He told a member Edison's staff sitting with him that "when Miss Case stops and the phonograph sings alone, I'll tap your foot; when Miss Case starts to sing in unison with the instrument, I'll press your foot again." Although the singer sang with the instrument twice and twice let the phonograph "sing" alone, the man never signaled that he detected a change. The story concludes: "Upon completion of the song, the artist arose and said, 'She sang with it all the time, you can't fool me.' Even though the others in the box assured the artist he was wrong, he steadfastly refused to believe it."

Robert Harley -- Thu, 07/24/2008 - 08:07

That would appear to be the first instance of a blind listening test. It's interesting that he didn't hear any difference.

returnstackerror -- Mon, 08/11/2008 - 19:36

tmartin states "I'm sure this doesn't work perfectly, but I have to say I don't think the imperfections are mainly due to brand/price bias. That's because, as one of the participants in this system, I have heard a lot of the equipment that others have reviewed and I hear what others describe pretty easily and regularly. Maybe I have the same brand/price biases, but then it is weird that I agree with my colleagues descriptions, but not often with their thinking of how they'd spend their money. "

This thread has a large focus on a listener being swayed by the cost of/maker of components under evaluation.

Does this also subconsciously happen with reviewers, either as the "first off the block" with a review of a component or those others that follow.

Are reviews subconsciously influenced by what else has been professional written about a component they themselves are reviewing?

Peter

Robert Harley -- Tue, 08/12/2008 - 11:09

Peter asks if the strong correlation between sonic descriptions of products among the world's reviewers is the result of the first reviewer influencing the opinions of subsequent reviewers.

In many cases, reviews of new products are effectively published simultaneously; the publishing lead time dictates that when one review appears in print and another review of the same product is out two months later, the second review would have been finished by the time the first review was published. This is true for print magazines, not online reports.

I can't speak for all reviewers of course, but in my own listening evaluations I believe that my perceptions of a product's sound vastly overwhelm anything I've heard or read about the product previously.

Sometimes you'll find similar sonic descriptions, but different judgments over whether that specific sonic character is good or bad.

franktren -- Thu, 09/18/2008 - 22:00

Robert Harley says, basically, that to say one can describe the characteristics based on measurements is absurd (I am paraphrasing here). And then notes Keith Johnson's work.
Not so fast, Robert. I seem to remember that Richard Heyser did some ground-breaking work doing just that.
I believe it is important to distinguish between two qualities as we debate this issue.
On the one hand, Robert is correct if he defines the outcome as describing how the subject at hand sounded to him, in his environment, using his sources and his equipment. Thus, he is describing, or attempting to describe, how his ear-brain perceives the subject. On the other hand, the outcome of such work as Heyser and Johnson isn't that, at least for Heyser. He attempted to describe certain particular characteristics that the subject would exhibit based on his testing. Quite different. Many of the characteristics he forecast turned out to be deficiencies of the subject.
This is not to discredit the opinions and/or the methods of either side.
To me, it simply means that to fully characterize a sound, produced by a transducer, we should, or at least could benefit from using BOTH.
That is my opinion.

Robert Harley -- Fri, 09/19/2008 - 10:10

I have never rejected the role of measurement in audio. But because so many readers have that perception, I take responsibility for giving that impression in my writings on the subject. Rather, my position is that listening is a valid and vital method of determining audio equipment quality. My writings have largely been in response to those who think THD, IMD, frequency response, and other standard measurement data can completely characterize the performance of an audio component.

Incidentally, I'm a big fan of Richard Heyser, as you can see from the number of quotes of his I've incorporated in my AES paper "The Role of Critical Listening in Evaluating Audio Equipment Quality." You can read the entire paper here:

Andy Simpson -- Sat, 09/20/2008 - 03:52

robert_harley6 wrote:The quote from the Edison demonstrations is interesting. Every generation has believed that its audio technology achieved absolute realism. Another quote from that period is Metropolitan Opera singer Anna Case who said "Much to my astonishment, I was unable to tell the difference between my own voice, and Mr. Edison's recreation of it. (1919).

The nature of the phenomenon is of great interest here.

If the opera singer was genuinely unable to tell the difference between herself singing and a reproduction, I think we can dismiss her testimony or else turn it against the organisers of the test.

In the case of the demonstration you mention in the 1960s, do you have any more details?

There are other cases of 'fooling' the audience, which usually involve serious expectation bias, without which the tests would fall over.

Andy

www.SimpsonMicrophones.com - Next Generation Microphones

www.SimpsonMicrophonesArchives.com/Early_Music_A.mp3 - www.SimpsonMicrophonesArchives.com/Early_Music_B.mp3

Robert Harley -- Mon, 09/22/2008 - 14:15

I don't have the details of the 1960's Acoustic Research test, but remember reading that a reviewer of the day wrote (in Audio, I believe) that he couldn't distinguish between the sound of the speakers and of the live performance (a chamber group).

Andy Simpson -- Wed, 09/24/2008 - 07:55

robert_harley6 wrote:I don't have the details of the 1960's Acoustic Research test, but remember reading that a reviewer of the day wrote (in Audio, I believe) that he couldn't distinguish between the sound of the speakers and of the live performance (a chamber group).

I wonder if in this case we are talking about marketing, expectation bias or genuine results.

It would not be hard to bias the test in favour of such a result - for example, by locating the subject at a great distance.

However, having done recent tests with chamber groups (see mp3s below), I can't imagine those AR speakers generating the basic SPL or the crest-factor required, particularly if the group included a soprano or any other loud voice, particularly from such a distance as would be required.