Just before the sessions began, Paul Barton explained that we would be hearing four speaker systems whose volume levels had been matched (at a specific midrange frequency), where the identities of the speakers would be unknown to us and to the NRC staff member running the experiment. We would listen to a mix of musical selections, with the experimenter switching between the speakers in random order as the music played (this is where those aforementioned ID lights come in handy), and our job was to rate all four speakers using the parameters above. Toward the end of the listening session, we would be allowed some time where the listeners themselves could control the speaker switcher, if, for example” they wished to compare on specific speaker versus another (say “2” vs. “4”). Listeners (and remember, these are journalists and some PSB-related PR/marketing guys we’re talking about) were told to try to remain expressionless and not to communicate either verbally or non-verbally while the listening sessions were in progress. We tried.
Barton further explained that, in the first of our blind listening tests, we would hear three speakers under comparison to one another, plus a fourth “anchor” speaker whose performance was decidedly off pace relative to the other three. Then, in the second test session, we would hear four speakers under evaluation—some carried forward from the first session, but some that were not.
In three words, I found the tests “revealing,” “demanding,” and “stressful”.
Revealing: The blind screen really forces you to ask, “What are the actual differences between these speakers” and, “How big are the differences, really?” In a way, it’s thoroughly refreshing to hear products without any biases regarding size, shape, price, configuration, or presumed design pedigree. The good news is that this is a true “what you hear is what you get” listening experience—one that, in a sense, invites listeners to free their minds of preconceptions and simply observe what is. I’m onboard with that idea.
Demanding: Unlike leisurely in-home listening tests, where you can use familiar musical materials in diagnostic ways to hear how one speaker handles a set of specific musical passages vs. another, here the material was not necessarily familiar and the randomized switching did not necessarily lend itself to learning as much as possible about the speakers under test. There was, for me, the pressure of needing to come up with accurate ratings in a very short span of time, and without making nearly as many specific comparisons as I might in a typical Playback review context. So, the demand was to “get the judgment” right, right away, and with incomplete sonic data.
I found that I could tell a lot about the speaker’s apparent frequency balance characteristics (and about the smoothness of their response curves), and could also learn a fair amount about their apparent resolution, transient speed, and textural characteristics. However, I could learn little about their imaging and full-fledged spatial characteristics, largely because—as I learned after the first session was completed—we were listening to single samples of all loudspeakers playing in mono. (Am I the only guy who feels this is a strange way to evaluate a product that will almost always be heard in stereo? Just asking…). Why mono? According to NRC, listening to a single speaker in mono gives more consistent test results and tends (or so it is theorized) to avoid various masking effects that a stereophonic presentation might entail. Still, I retain a healthy skepticism…
Stressful: I suppose audiophiles in general and audio journalist in particular take pride in their listening and evaluation skills, so that a double-blind listening session introduces stressors galore. Questions such as, “What if I’ve got garbage for ears?” or “What if my powers of discernment aren’t as keen as I think they are?” go racing through your head, exacerbated by the constant time pressures at hand (not to mention an utter absence of traditional left/right imaging cues to work with).
Finally, I found that the available listening seats were far from similar (although in theory they are supposed to yield fairly consistent listening test results). The room is designed so that four listeners can take part in tests at once, with two chairs placed in the front, and two more chairs directly behind them. In the first session, I was in the left/front seat and for the second session I was in the right rear seat. I found it was much harder to make critical assessments about speaker performance from the right/rear seat where I was seated directly behind another listener—a listening position I would rarely if ever encounter in my home. Still, I did the best I could, and waited to learn results later.