Blind Listening Tests are Flawed: An Editorial
Robert Harley -- Wed, 05/28/2008 - 16:18
The following is my editorial from The Absolute Sound Issue 183 (not yet published) on blind listening tests.
The Blind (Mis-) Leading the Blind
Every few years, the results of some blind listening test are announced that purportedly “prove” an absurd conclusion. These tests, ironically, say more about the flaws inherent in blind listening tests than about the phenomena in question.
The latest in this long history is a double-blind test that, the authors conclude, demonstrates that 44.1kHz/16-bit digital audio is indistinguishable from high-resolution digital. Note the word “indistinguishable.” The authors aren’t saying that high-res digital might sound a little different from Red Book CD but is no better. Or that high-res digital is only slightly better and not worth the additional cost. Rather, they reached the rather startling conclusion that CD-quality audio sounds exactly the same as 96kHz/24-bit PCM and DSD, the encoding scheme used in SACD. That is, under double-blind test conditions, 60 expert listeners over 554 trials couldn’t hear any differences between CD, SACD, and 96/24. The study was published in the September, 2007 Journal of the Audio Engineering Society.
I contend that such tests are an indictment of blind listening tests in general because of the patently absurd conclusions to which they lead. A notable example is the blind listening test conducted by Stereo Review that concluded that a pair of Mark Levinson monoblocks, an output-transformerless tubed amplifier, and a $220 Pioneer receiver were all sonically identical. (“Do All Amplifiers Sound the Same?” published in the January, 1987 issue.)
Most such tests, including this new CD vs. high-res comparison, are performed not by disinterested experimenters on a quest for the truth but by partisan hacks on a mission to discredit audiophiles. But blind listening tests lead to the wrong conclusions even when the experimenters’ motives are pure. A good example is the listening tests conducted by Swedish Radio (analogous to the BBC) to decide whether one of the low-bit-rate codecs under consideration by the European Broadcast Union was good enough to replace FM broadcasting in Europe.
Swedish Radio developed an elaborate listening methodology called “double-blind, triple-stimulus, hidden-reference.” A “subject” (listener) would hear three “objects” (musical presentations); presentation A was always the unprocessed signal, with the listener required to identify if presentation B or C had been processed through the codec.
The test involved 60 “expert” listeners spanning 20,000 evaluations over a period of two years. Swedish Radio announced in 1991 that it had narrowed the field to two codecs, and that “both codecs have now reached a level of performance where they fulfill the EBU requirements for a distribution codec.” In other words, Swedish Radio said the codec was good enough to replace analog FM broadcasts in Europe. This decision was based on data gathered during the 20,000 “double-blind, triple-stimulus, hidden-reference” listening trials. (The listening-test methodology and statistical analysis are documented in detail in “Subjective Assessments on Low Bit-Rate Audio Codecs,” by C. Grewin and T. Rydén, published in the proceedings of the 10th International Audio Engineering Society Conference, “Images of Audio.”)
After announcing its decision, Swedish Radio sent a tape of music processed by the selected codec to the late Bart Locanthi, an acknowledged expert in digital audio and chairman of an ad hoc committee formed to independently evaluate low-bit rate codecs. Using the same non-blind observational-listening techniques that audiophiles routinely use to evaluate sound quality, Locanthi instantly identified an artifact of the codec. After Locanthi informed Swedish Radio of the artifact (an idle tone at 1.5kHz), listeners at Swedish Radio also instantly heard the distortion. (Locanthi’s account of the episode is documented in an audio recording played at workshop on low-bit-rate codecs at the 91st AES convention.)
How is it possible that a single listener, using non-blind observational listening techniques, was able to discover—in less than ten minutes—a distortion that escaped the scrutiny of 60 expert listeners, 20,000 trials conducted over a two-year period, and elaborate “double-blind, triple-stimulus, hidden-reference” methodology, and sophisticated statistical analysis?
The answer is that blind listening tests fundamentally distort the listening process and are worthless in determining the audibility of a certain phenomenon.
As exemplified by yet another reader letter published in this issue, many people naively assume that blind listening tests are somehow more rigorous and honest than the “single-presentation” observational listening protocols practiced in product reviewing. There’s a common misperception that the undeniable value of blind studies of new drugs, for example, automatically confers utility on blind listening tests.
I’ve thought quite a bit about this subject, and written what I hope is a fairly reasoned and in-depth analysis of why blind listening tests are flawed. This analysis is part of a larger statement on critical listening and the conflict between audio “subjectivists” and “objectivists,” which I presented in a paper to the Audio Engineering Society entitled “The Role of Critical Listening in Evaluating Audio Equipment Quality.” You can read the entire paper here http://www.avguide.com/news/2008/05/28/the-role-of-critical-listening-in-evaluating-audio-equipment-quality/. I invite readers to comment on the paper, and discuss blind listening tests, on a special new Forum on AVguide.com. The Forum, called “Evaluation, Testing, Measurement, and Perception,” will explore how to evaluate products, how to report on that evaluation, and link that evaluation to real experience/value. I look forward to hearing your opinions and ideas.
Robert Harley
All content, design, and layout are Copyright © 1999 - 2011 NextScreen. All Rights Reserved.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
"I must say I am shocked about this so-called expert's lack of understanding around the concept of ABX testing."
>>>If what you say is true, he is off. I would have to read the article myself as you are not exactly trustworthy yourself
as we shall see. I have seen this kind of rethoric before and found the summary to be misleading or outright false.
Hopefully it is true in your case. (Folks, you might read my previous posts on this forum page for some additional
scientific information on the subject.)
1) "It's completely scientific and relevant to the situation in audio where there is a very transitory memory and equally as good as just comparing A with B. "
>>>>>A) So you state we have "very transitory memory" (audio) yet claim an ABX test, which takes much longer is accurate. So which is true, very transitory, or much longer? You cannot claim science and then attack science Kel. You cannot claim both to be true Kel.
>>>>>B) "and equally good as just comparing A with B". Quite a claim considering the statement is so vague. How many ABs? What SPL? Did you keep track of each AB? What is the room acoustics like? What selections did you use?
Sure problems can happen in any test or audition. Unfortunately you are attempting to prove, although very poorly, that ABX testing is and of itself
scientific and accurate.
"the test is more straightforward to set up and interpret.
I have studied statistical analysis in depth during my extensive experience and I just find that discrediting ABX as kind of voodoo shows an uncomplete conceptual understanding of science and only serves as to discredit further the snake-oil peddlars that don't like the idea that ABX doest's support their misconceptions."
>>>>>Unfortunately Kel, you only address the abx testing after the total count/ABs are tallied and the numbers are crunched. That is typical of many websites supporting audio dbt/abx testing. Unfortunately, you, the sites, fail to mention any information concerning how the test is performed, are the ABers being influenced in their individual decisions, what information is compiled etc before the numbers are crunched.
And you have the gall to call yourself scientific while demeaning others as unscientific? What is so misleading is how you and your friends above claim science and yet can't present an ounce of science.
You have
1) already contradicted yourself at least once,
2) provided a generalize mush comment with no meaning, yet make generalized "scientific" claims from that mush statement, and now
3) only present a partial, after the total ABs and numbers are crunched.
The trick to the test folks, is before the numbers are crunched. But you will never see anything in detail mentioned,
maybe a generalized statement like "we have referees, anonymous of course" who check things, ask questions etc.
"You simply just need to say how confident X matches the previous sample heard in the cases of first listening to A then B, with X in between in both cases."
>>>>>See my above comments as to why this statement is so inaccurate. Of course there are more.
Remember that after a few ABs, according to you, audio memory is worthless. So one's responses becomes a 50/50 guess.
Folks, you have just seen an example typically used to justify audio dbt/abx testing, and what
to look out for. We have seen from his post.
1) He generalizes what other's say. Is it accurate?
2) He contradicts himself, which is typical of other "scientists" on other forums
2) Only partial science is presented, which of course bolsters his position.
(Others have been known to purposefully present only partial information to
intentionally mislead you. So beware that all the science is presented. I am not accusing Kel per se as it could be ignorance.)
In conclusion, I think we all know the snake oilers, the pet rockers, who sell rocks to
improve sound etc. We all know them.
What is insidious are those who claim to be
scientists and who have been witnessed red handed (on several forums) of falsifying data,
who claim to be scientists but have conflicts of interest.
altering large scientific tests to discredit them, attacking competitors of their friend's Co. They use
science as a cover to push their wares, their friends wares, or their employers/industry wares.
Folks, you have a template, provided by these gentlemen, in which to check out other "scientific" false agruments and audio tests on other
forums.
Cheers.
Steve, thanks for your post. But, again the question I believe for a person who just likes to hear good sounding music is not weather there is any scientific difference. The question isn't even weather you can hear the difference. The question is, "does it sound any better"! ? Assuming you are listening through a relatively good set of speakers (i have a pair of dahlquist dq 20's that ive had since 1989) to my ears the answer is no. 96/24 Wav and 128 mp3 do sound the same. Coat hangers and hi end cables sound the same. sacd and cd sound the same. Radioshack and high end directional rca's sound the same. when I say they sound the same I mean that any difference I think that I can hear is sooooo negligible as to not justify the huge difference in cost to hear it. And even if i can, does it sound any better? This is not to in any way trivialize what you do. It's just that for most of us the only real huge difference is in the loudspeaker.
Steve,
It would be great if you could share with us how you would conduct a blind listening test that gives scientifically sound results, or are you of the opinion that it is impossible to do?
Best,
ev
Keladrin -- Wed, 04/18/2012 - 03:39
Hear Hear!
Kevin
Your comments are appreciated but the subject is concerning blind listening testing accuracy.
Check the top of the page for the topic. As such I am on topic.
Cheers.
I'm sorry. The auto email I got said you were responding to my post. However I think I am on topic also if you look at my original post a few replies up the list. The conversation was an offshoot of the original question, and not a new topic since hearing a difference is an arguement that seems to have gone for decades. Just curious. Do you hear a difference when you use highend cables vs coat hangers etc in an abx test? If so, how do you determine which is "better" ? Also if I can't hear a difference with the only ears I have then that validates the test for me.
Hi Steve,
I ask you this. You say in ABX testing you have to be very careful in that there is no clue leading to bias - but bias would lead to a false positive, not a negative result! Do you understand this? Ok then just how much bias is introduced if you suddenly introduce visibility of the nameplates as in conventional trials. Do you think that if you did this you would nearly always get a positive result? You would be right and this itself would support the fact that knowing at what you are listening to is the overriding factor skewing results in any so called conventional 'test'. It’s quite funny that you pick such microscopic faults with the scientific approach but seem quite happy with the alternative, which to all intents and purposes amounts to totally unregulated subjective hearsay. That is the serious flaw in your argument - you can't just knit pick without looking at the alternative or offering a better approach (Golden Ear Reviews no doubt, where you rely on a sonic memory of several hours).
Most doubters of any kind of professional testing just dig a hole for themselves like this in analysing the test protocol to the n'th degree, so discrediting your altenative anti-science 'test' approach even more.
Kevin
" You say in ABX testing you have to be very careful in that there is no clue leading to bias - but bias would lead to a false positive, not a negative result! Do you understand this?"
>>>>Unfortunately your comment is incorrect as we will see below, from your own words. If during the test, variables, which you again failed to address, are not accounted for, the responses are skewed towards No sonic difference.
Secondly you address seeing name plates as if we are addressing sighted testing. The topic is audio dbt/abx accuracy.
Let's not go the misdirection route Kel.
" It’s quite funny that you pick such microscopic faults with the scientific approach but seem quite happy with the alternative, which to all intents and purposes amounts to totally unregulated subjective hearsay. "
>>>>Really? We should not be using a deviation of the bait and switch tatic, exaggeration comparison; Microscopic VS Totally Unregulated
Subjective Hearsay. If you check my previous posts, this is what EV stated,
3-24-12, page 4: "I also enjoy Steve S's commitment to making sure tests are done in a meticulous disciplined way. "
>>>Quite different than what you, Kel, just attempted to push on us. I also don't see you addressing with science, just
generic general comments.
"Microsopic faults"? Folks, check my post, page 4, 4/17/12, 13:59 for the so called "microsopic" faults. The "nitpicking" as he now calls it. Notice
especially the contradiction(s) Kel posted, which represents the typical so called "Scientific" group's argument you see posted
on the forums. But there is more.
"That is the serious flaw in your argument - you can't just knit pick without looking at the alternative or offering a better approach (Golden Ear Reviews no doubt, where you rely on a sonic memory of several hours). "
>>>>>Your contradicton and exaggeration comparison, and more demonstrates otherwise Kel. Since, as you put it, audio memory is a "very transitory memory" then most responses from ABX test subjects are nothing more than a flip of the coin, which is approx 50/50. So the response total is inaccurate, skewed, the result/conclusion is also skewed/rigged to No sonic difference. Thanks for providing the valuable information.
As one can see, Kel not only has contradicted himself, but also attempted to mischaracterize his mistake by calling it "microscopic" and me "nitpicking". Not surprising since his information demonstrates that the audio dbt/abx tests are skewed, if not outright rigged towards no sonic difference.
Now you see why Kel is calling his contradiction(s) "microscopic" and me as "nitpicking".
Cheers.
Steve,
I think for us to better understand the significance of the variables you are concerned about, it would be helpful to get more specific information about them. In your previous posts you have referred to the following variables:
sight (go blind)
manufacturer
cochlea fatigue
habituation to stimuli
spl level
time duration
ABs per individual
Room acoustics
In the interest of improving our comparative listening skills, This is what I would REALLY REALLY like to know:
In your opinion/experience:
Is it better to do listening tests sighted or blind?
Is it better to know the manufacturer or not?
how does one avoid cochlea fatigue when doing listening tests?
how does one avoid habituation to stimuli when doing listening tests?
what is the best SPL level when doing listening tests?
what is the best time duration when doing listening tests?
How many ABs per individual is best when doing listening tests?
at what point do the inaccuracies associated with room acoustics begin desensitizing a listening test?
Thank you for your input,
ev
It's important to undertake audio testing with a degree of intelligence and common sense. My comment about nitpicking is basically saying you are getting too bogged down in the particular test implementation and not looking at the bigger picture. The purpose of a blind trial (or ABX) is to ascertain with a reasonable degree of significance (or not) that differences can be heard and the whole point of such a test is that you just can't ascertain this from a standard review or one off observation that fails to 1. hide nameplates and 2. fails to match levels.
To mention the alternative to such a test is not misdirection at all but putting the text in context with its intended use. Your problem (and those that continually analysise test protocols until they think they have found some fault) is that you talk yourself into a corner in pedantic pseudo-scientifically analysing test protocols until you have convinced yourself that they are not telling you anything useful. For example your misconception that ABX testing is invalid because you think it is skwewed to give false negatives due to audio memory. So how do you propose to address this issue - use a conventional test where you listen for a few days when you say yourself that audio memory is so volatile (and are right with this) This is an example of you talking yourself into a corner and none of your arguements show any logic I'm afraid, just embarrissingly misquoted, twisted and misunderstood snippets of my original post. Please present your logical argument against blind trials more clearly, thanks.
Kevin
Hi Steve,
Yes you have already said you don't or won't understand my posts. It's a shame you have backed out so quickly as I (and many other here) really wanted to hear your example of your perfect test - does it involve peering ath the nameplates by any chance? What a joke.
I will make it very simple for you. In compative testing what special insights are gained by 1. Not hiding the nameplates? and 2. Not matching levels?
If you can answer these questions with any kind of sanity I will concede that your unblind test is the least flawed. Can't put it simpler (or fairer) than that!
Those reading these posts will probably already have come to the conclusion that those who oppose blind testing the strongest fear them the most.
Kevin
Steve, I am not asking you to reveal any trade secrets. I am going to intentionally stay away from any comment regarding your issues with Kel. I am simply asking you how, if abx testing is not a valid method, you suggest I go about evaluating components. You said you use listening tests. That means that the human ear is what you use to evaluate the equipment. What I have seen from going back and reading all of your posts to make sure im not missing anything is that you never seem to respond to direct questions put to you. EV in an earlier post asked you some very specific questions about how we would go about improving our listening skills. He asked:
In the interest of improving our comparative listening skills, This is what I would REALLY REALLY like to know:
In your opinion/experience:
Is it better to do listening tests sighted or blind?
Is it better to know the manufacturer or not?
how does one avoid cochlea fatigue when doing listening tests?
how does one avoid habituation to stimuli when doing listening tests?
what is the best SPL level when doing listening tests?
what is the best time duration when doing listening tests?
How many ABs per individual is best when doing listening tests?
at what point do the inaccuracies associated with room acoustics begin desensitizing a listening test
You have presented yourself as an expert and I accept that. Instead of focusing on Kel would you PLEASE help us here? Some of us have a genuine interest in trying to figure this stuff out and the input from an expert like yourself would be very valuable. You said :
As far as how I test, I am not stupid enough to give it out for free so my
competitors can copy. However, I can divulge I have no sight problems and
I have no manufacturer's problems since it is my component under test.
I have no idea what that is supposed to mean and doesnt even remotely answer my question Like EV, I asked you a couple of sincere specific questions, as an expert in your field, and again, instead of responding you made some reference to Kel's family. Are you saying that if I came into your shop you wouldn't help me by telling me how I could better evaluate your system so I could compare it to others before investing my hard earned dollars? If I'm going to spend $180 for 2 meters of RCA cables you are gonna have to tell/show me how to hear the difference because it's been my experience that there is none, but I am willing to spend the money if you could demonstrate the difference.
Instead of investigating Kels background stay on topic and help us out here. I for one don't care how old Kel is. His opinion is just as welcome as yours. Like EV said, help us improve our listening skills. PLEASE! As an expert, if blind tests are flawed how would I evaluate an SAS component?
Oh dear, I think we have just seen Steve's true colours, No that isn't my profile, sorry Steve.
Ramaroodle I will answer your/EV's questions:
Is it better to do listening tests sighted or blind? Blind
Is it better to know the manufacturer or not? Definitely not
how does one avoid cochlea fatigue when doing listening tests? Allow the user to listen at their leisure - they are infact in charge of duration, switching etc (within the protocol) Cochlea fatigue infact applies to all tests (blind or not) so is not a major consideration in comparing the two
how does one avoid habituation to stimuli when doing listening tests? As above, certainly allow a relaxed environment with rest periods
what is the best SPL level when doing listening tests? A level the amplifier can easily accommodate and is comfortable listening. It is important that levels are matched for obvious reasons
what is the best time duration when doing listening tests? Let the user decide this (within constraints of protocol). I would say maximum period of half an hour per listen, minimum period no limit
How many ABs per individual is best when doing listening tests? the most the better, at least 10. The number effects the confidence level of the result, for eaxmple you won't achieve 95% confidence (the agreed standard) with 2 tests.
at what point do the inaccuracies associated with room acoustics begin desensitizing a listening test - this will effect all tests. Room resonances should be minimised, certainly to a degree less than that usually experienced in a typical domestic situation
Kevin
All,
The forums are designed to facilitate diplomatic discussion about the audio topics we all enjoy. That being said, threats toward other users and their families is utterly unacceptable and will not be tolerated.
-AVguide Staff
A bit more detail provided here:
'How many ABs per individual is best when doing listening tests?' 10 is the minimum to establish 95% level (9 out of 10) but more is better. Probably best to limit to below 25 per test period. Obviously fatigue will set in with huge numbers of tests without a break. For confidence level table see below:
Number of trials 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Minimum number correct 9 9 10 10 11 12 12 13 13 14 15 16 16 16 17 18
Kevin
I hope it's not too late to chime in on this important topic even though the thread is four years old. I don't pretend to be an expert on blind vs. sighted listening tests, but I do have some training and experience in the area of designing and evaluating research. I have a PhD in experimental psychology and I teach critical evaluation of biomedical research articles (for example, randomized, blinded controlled trials comparing the effectiveness of drugs for disease treatment) at a medical school. And I'm an audiophile and amateur speaker builder. Here are a few points that come to mind.
1. It's been established through speaker listening tests at the Harmon Kardon research labs that the appearance of speakers can influence listeners' ratings of sound quality. That is, large, impressive looking speakers may receive higher ratings in sighted testing than in blind testing. In other words, sighted testing can easily introduce bias due to listener expectations based on appearance, cost, etc. So you can make a case that first listening impressions should be formed "blind" to reduce these sorts of biases. I make this point to illustrate the general value of blind testing and an established problem with sighted testing; this point is not a justification for the specific method of A/B blind testing.
2. A poorly designed and conducted experiment invalidates itself, not the general research method being used. Mr. Harley wrote in 2008 about "the flaws inherent in blind listening tests" and, finding the results of a study published in the AES journal unacceptable, stated that "such tests are an indictment of blind listening tests in general because of the patently absurd conclusions to which they lead." Not so, Mr. Harley (my apologies in advance if you've changed your position since 2008; I didn't search for more recent writing on this topic). A better approach would be to look for methodological flaws in the studies. There are many possible reasons why a study would result in a Type 2 error (failing to detect a difference that's actually there). Finding those reasons and correcting them advances science. Blanket dismissal of what appears to be an applicable research method doesn't advance science.
3. An experiment should be designed with an explicit hypothesis in mind, and with an estimate of the effect size (e.g., the magnitude of the difference) that the experimenter wants to detect at a statistically significant level. The estimate of effect size is necessary in order to select the number of data points (e.g., number of participants and number of trials) that are needed to have a strong chance (usually set at 80% or 90%) to detect a difference of the specified size or larger (if it's actually there) at a statistically significant level. This is called a "power analysis." I doubt that most A/B listening experiments have been set up with an explicit hypothesis and any sort of power analysis, so there's a chance that they miss real differences. However, if a listening test required 100 golden ear listeners and 1000 trials to detect a difference at a statistically significant level, that would be a very small absolute difference that might not be of much practical importance to most listeners.
I'd better stop there before I wear out my welcome. Thanks for the opportunity to participate.
Doug
Hi dmann,
Some shrewd observations and I agree with them.
If you look again at Mr Harleys original quote 'Every few years, the results of some blind listening test are announced that purportedly “prove” an absurd conclusion.'
The major flaws are in the preconceptions that this statement wrongly implies. First of all the blind test does not attempt to 'prove' a negative result. Infact a negative cannot be proved as such, only the absense of a positive. Like you say the test can only establish whether within the experimental variables a difference can statistically significantly be heard or not. A neagtive result has to be taken in context with the test objective.
It can only take one subject to 'dispove' the negative, however. The validity of such a test rests on this fact, ie if a difference is discernable then it should not be much trouble to detect it. The negative result is validated by the fact that the opposite 'proof' has not been established and is a default scenario.
As to whether the conclusion is 'absurd' this is wrongly based on a preconception on the outcome of the test. It is up to the best test to establish whether a conception is absurd or not. It is up to the designer to come up with the objective behind the test and best test available. In other words to say such a test 'purportedly proves an absurd conclusion' is an absurd statement in itself because:
1. It prejudges the outcome of the test. A result can only be judged 'absurd' if it disagrees with the result of some previous more valid test, unless you are relying on (gross) misconception instead of testing (which is more likely the case here).
2. You cannot 'prove' a negative conclusion, a negative result is the best available implicator of a negative conclusion in context with the test objective
To quote the voice from the back of the room in the Firesign Theater's "Don't Crush That Dwarf, Hand Me the Pliers":
"That's metapheesically absurd, mon, how can I know what you hear?"
I posted a much more extensive comment yesterday which was summarily eaten by Mozilla 18.0
First, I do note well the word "indistinguishible". It's an absolute statement, so my tendency is to treat it with suspicion. Not indistinguishible by some people or by most people or on most program material, but flat out indistinguishible. It doesn't, for example, account for the possibility that 1 in 100 listeners, or one in 1000 may readily and consistently distinguish. Some humans, for example, have perfect pitch, enabling them to distinguish a 435 hZ tone heard yesterday from a 440 hZ tone heard today.
An absurd conclusion, though, it most definitely is not. Although the human ear is an incredibly sensitive organ, it does have limits. Even the most golden-eared among us will at some point not be able to distinguish between two physically different sounds. The physical difference between Redbook sampled and higher resolution signals can be distinguished by appropriate instrumentation. The conclusion may not jive with your experience, or with mine, but it is not absurd.
What is absurd is the converse: perceived differences in sounds that are physically identical. These days the magazine reviewers who hear six impossible things before breakfast want us to believe boutique power cords make night and day differences, or fancy speaker cable stands that relieve static charges. If we want the best possible sound, we are told, we have to pony up and not ask questions. I call such an attitude unscientific, not just because the conclusions violate accepted science, which they do, but because anyone honestly reaching them with an even halfway scientific bent would lie awake nights for wondering why.
I have never heard an explanation for why three feet of power cord should make a difference for all the bad sounding Romex ahead of it that didn't have my BS meter slamming on the upscale. I would love to have some of these reviewers held accountable without wiggling out with some Heisenberg uncertainty principle-esque argument as to why blind tests are invalid.
I fully acknowledge blind testing methodology can lead one hideously astray, as scientists have been doing occasionally for centuries. But for results that are reproducible I maintain it is the best we have.
p.s. I would love to read your full article but the link doesn't get me there. Do I need a paid subscription or something?