Preselection bias/ Is a no review = bad review?

David Matz -- Sat, 02/06/2010 - 16:49

The thread in the Speaker forum on the Magico V2 speaker has generated a lot of interested comments. Rather than hijacking that thread, I thought I would start a new one.

In that thread, Mr. Sircom has made an interesting comment regarding how rather than give a bad review if the component does not match a reviewer's taste, the reviewer would pass the component on to a colleague or refuse to write a review. Can one then conclude that no reviews means the product is awful or mediocre?

Also,I get an impression of a hoard mentality among reviewers. At shows, I wish the audio experts would go around blind folded and gagged until they write up and post their show impressions. As far as the actual reviews go, I remember when the latest top of the line Theil speaker came out, it took 2 years to get a review. Wes Phillips, if I remember correctly, then raved about it. TAS followed with a positive review. I think something similar happened with YG. Phillips raved about their top of the line speaker. Mr. Harley reviewed the Kipod 6-8 months later. Does TAS feel comfortable coming out first?

Thanks
Thanks

Mr Plus -- Sun, 02/07/2010 - 07:03

Like most things in life, it's not so clear-cut. A product that gets no reviews is either 'not available for review', has not gotten round to be reviewed, has slipped through our respective fingers or is simply not relevant for our readership. If a product is 'not available for review', it might mean the product is below par, that the magazine and manufacturer have a difference of opinion, or simply that the company doesn't give products out for review. This need not be malign; Harbeth recently announced a worldwide halt on reviews because its order books are filled to bursting and yet another review of its products would mean year-long waiting lists.

I disagree about a hoard mentality among reviewers (well, I would, wouldn't I?). When it comes to shows, if you took a survey of 100 show attendees (whether they were in the 'trade', a keen audiophile or 'just passing thru'), chances are you would end up with five or six 'good' rooms noted pretty much universally and about a dozen that were regularly mentioned. We sometimes compare notes mid-show to see what's worth checking out (in part because if you try to visit every room in a show, you'll spend less than 30 seconds in each room), but we don't wait for a good report and simply agree with that first report. The only way these things do not read like a pack agreeing with itself is to either give individual reviewers specific aspects of the show to comment on (and getting a group of reviewers do do anything in a coordinated manner is like herding cats) or to produce factual reports on new products and nothing else. This last doesn't sit right with those who read show reports, who instead seem to prefer the subjective 'flavor' of a show they were unable to attend every bit as much as the objective coverage of any new products launched there.

Finally, as to reviews hitting the pages in staggered form, there are two good reasons for this. A company might get just two chances to put a product into print in the US; once in TAS, once in Stereophile. If the reviews hit the streets within a month of one another for a product with a life cycle of maybe five years or more, that gets all your print coverage out the way in very short order. Most companies prefer a roll-out of review - gap - review, to keep the product fresh in the audiophile mind. Another issue is that with very high end products there might only be one review/demo sample available at product launch. An importer of something like the top MBL loudspeakers for example isn't going to have dozens of demonstrator pairs to push around the country, and that means magazines and dealers alike might need to take it in turns to listen to a product. So this might mean TAS gets the first review, then it goes on the road, then Positive Feedback, then on the road again, then Stereophile and so on. That could means as long as a year between reviews.

Does a magazine feel comfortable coming out with a review first - of course! If it's a hot product and we get first 'dibs' on it, that scoop not only gives us the 'feather in the cap/to boldly go where no magazine has gone before/ner ner nyer ner ner' feeling, but it can have a knock on effect on magazine sales.

Alan Sircom
Editor, Hi-Fi Plus Magazine
London, England
editor [at] hifiplus [dot] com

David Matz -- Fri, 02/12/2010 - 10:18

Mr. Sircom,

Thanks for replying. I agree with your point about staggering the reviews - it makes a lot of sense. I also agree that if it's a hyped product from ARC or Wilson, everyone is all over it. I guess if everyone is interested, and they are independent enough, it's a good thing. However, it could also be a hoard chasing the tulips in the Netherlands or tech stocks or real estate.

However, this site lists many products. For each product - amp, preamp, speaker, digital, etc., there is a platform such as Reference, "Top Class", "Low class", etc.

For example, there are very few reviews of Burmester products or VTL reference class. There may be one here or there. From a decision making stand point, there is just not enough data to warrant an audition. If there is a single review from an "expert", one cannot rationally conclude anything about the product, unless one has consistently experienced a correlation between one's tastes and the expert's suggestions. (If one believes experts are infallible, try asking three "great" doctors how they would deal with the same health issue or three Nobel laureates what should be done with the economy.)

So what's your advice to audiophiles trying to make a decision if the product or a platform is not reviewed? Thanks!

Tom Martin -- Sat, 02/13/2010 - 12:10

Unfortunately, I don't think you can conclude much from a "non-review". First of all there are vastly more products on the market than our staff can review. Sometimes a non-review just comes down to a pure math problem. This is compounded by our desire to spread reviews across price points. Second, particularly with regard to products introduced at shows, sometimes the lag from introduction to production is quite long. Some of that is down to the timing of CES: it happens once a year, so manufacturers with limited budgets need to get everything from "it'll ship next week" products to raw prototypes in front of dealers and press -- it is their one chance for 365 days. Thirdly, some product doesn't get reviewed because it didn't show well. Reviewers have to use limited information to figure what they want to propose for review. Now say a manufacturer had a prototype that the show personnel dropped from a forklift the day before the show. Might be a great product, but it was damaged slightly. Reviewer listens, isn't impressed, no review. Fourth, a product may have been heard or evaluated by one reviewer. Every reviewer has some orientation regarding what seems interesting. If the reviewer who saw the device misses the hook for the product it might not get reviewed. That doesn't mean it isn't worthy, OTOH we can't have every reviewer look at every product. Finally, the reviewer for a product may have a backlog.

I don't mean to make the process sound too chancy. Between shows, PR folks, discussions with manufacturers, discussions with dealers, discussions with GEC members, and working these forums there are a lot of ways to find interesting products. But not all those paths are the shortest distance between A and B.

CEO and Editorial Director, Nextscreen LLC

danielaparker -- Sun, 02/07/2010 - 18:12

Hope not :-) I happen to have a few things in my system at the moment that have received next to no reviews: a pair of Neeper Perfection One speakers (temporarily on loan from my dealer while I'm waiting for my graphite Amatis, having returned the red violin), a Transparent PowerIsolator 8, Transparent MusicLink Ultra interconnects, Transparent MusicWave Ultra speaker cables, and Transparent power cords. The Neepers sound pretty nice to my ear, look pretty nice too. No idea whether the Transparent products are any good, they're my dealer's recommendations, but they're not detracting from my enjoyment of the music. Transparent certainly doesn't get much mention in the main audio magazines, the only exception being TAS's recent favourable comments about their entry level products, and Transparent doesn't figure in any reviewer's system, as far as I can tell. I suppose once I have my system completely set up I might experiment with swapping in some more favoured products and see if I can tell the difference, but for now I can't be bothered.

-- Daniel

Tom Martin -- Sat, 02/13/2010 - 12:17

Another important point: not all manufacturers support the review process. Spectral for example, is well known not to want reviews. We had a rare Spectral review recently; prior to that our only Spectral review in the last 10 years resulted from being able to borrow the products from a cooperative owner. This doesn't mean that the products aren't good; some manufacturers want to avoid the demand spikes and valleys that sometimes surround reviews.

rwortman -- Fri, 03/12/2010 - 17:41

Hey, If you like your system you don't need the approval of an audio scribe to continue your enjoyment.

David Matz -- Sat, 03/13/2010 - 16:46

"Hey, If you like your system you don't need the approval of an audio scribe to continue your enjoyment."

Very true. I am sure there are folks out there who do need the approval. Audiogon would be half full if this were not so. As soon as there is a new product with a better review, sell it...

However, personally, I only find the reviews valuable as suggestions on what to audition. The more alternatives to consider, the better the outcome of the decision. Once I compile a list of alternatives, I trust my ears.