The truth about bike reviews

Tim over at Blue Collar MTB has let the cat out of the bag about bike reviews in magazines...or has he?  What is the truth about bike reviews in magazines? 

Here's an excerpt from Tim's post.

...over the last couple of months...I’ve run across a couple of bike reviews where the reviewer admitted to only taking the bike on a couple rides. I’ve heard rumors of different magazines getting paid to review a product...and other publications only reviewing products once the company has purchased advertising with them...

This was made clear to me recently when we were talking to a certain bike manufacturer about reviewing a bike for them and they would only allow us, at most, a month to review their bike. Unless I lived on this bike for that month there is no way we could give an honest review. So we turned them down. We’ve already had people tell us they have bought bikes based on our recommendation and the last thing we are going to do is compromise that trust just because some company says “that’s not how other people do it.”

I don't have a ton of knowledge about this  because I've only worked with one bike brand and reviews weren't a part of the plan.  But, I have thought about this issue and I'd love to know more about how it all works.  Are bike magazines being paid to review bikes?  I know advertising plays a part, but to what extent?  Whose responsibility is it to ensure objectivity and honesty in the review process?  The magazine editors?  The manufacturers?  The PR/marketing people?  The readers?

Comments

[Cory]

Bike reviews are useful only when they're taken with a grain of salt. They are especially useful narrowing down a bike choice but, when considering a new ride, I'd like to think that I (and others) will get solid advice from my local dealer.

For the review site that I run, we decided not to do reviews of full bikes for the very reason that you cite above -- it takes more than a few rides to form a decent opinion of a new bike.

What to do about it? Unfortunately there's not much that can be done. You'd like to think that publications (internet or print) would be forthcoming with their testing process but, of course, it doesn't always happen that way... Just set your standards, give your readers easy access to those standards, and go from there.

[Tim Jackson- Masiguy]

It's a slippery slope for manufacturers. The phrase "pay to play" didn't just materialize out of thin air. There is something behind it. In the past, I've had companies offer to do a review if you took out ad space. The implication is that the review will happen "if" you get the ad space. I haven't personally encountered an outright offer to do a review for money... yet. I'm certainly sure it happens.

To paraphrase, "it takes two to Tango"; magazines wouldn't be able to get away with selling reviews if manufacturers didn't buy them. There are some magazines, and I won't name any names, where the editorial content is a mirror-image of the ad space each month. Widget ad and widget review... always seems suspicious to me. In a perfect world magazines wouldn't have to rely solely on ad sales to stay afloat and then they would be able to spend more time being honest. Conversely though, manufacturers are in a tough position of looking for product evaluations in magazines with big circulation and that costs money too.

Personally, when I am looking at reviews I try to get a feel for how long it was tested, if they do not come out and say how long. A weekend review session doesn't tell me anything other than how it rides out of the box and the cables begin to stretch and the spokes settle. Still, the next time a magazine or site reviews one of my products on a tight time schedule and gives me a favorable review, I'm still going to use it in my marketing. Hey, it's a dog-eat-dog world out there...

Tim Jackson
Brand Manager
Masi Bicycles

[skinny]

I think the whole review thing can be done well and when it is done well you can easily tell.

I currently work as the marketing director for a fairly large knife company and formerly worked in the bike industry. We get offers all the time to run product "reviews" with an ad. Sales guy, "Buy an ad and we'll run info on your knife." This usually just means they will put an image and copy my press release almost word for word in the "edit" pages of the magazine.

At quick glance it looks good for the manufacturer. They get twice the page space for one low price. The magazine seems to benefit also, by filling more pages with ad and so called edit, making it look like a better value on the news stand. In the end it's all about the circulation. Sell more on the news stand, charge more for ads/"reviews" and make more money. So I may do those deals every so often just for the little bit of new exposure at a really reasonable rate. Many times these deals happen with magazine that are just starting out or have really low circulation.

I am almost 100% certain that this goes on in the bike industry. I had a couple offers like that but they were less frequent with the cycling magazines and occurred mostly with non vertical cycling pubs.

After a couple years of weeding out the good and bad, I now have a solid base of "good guys" (writers/reviewers/magazines) that I can trust to honestly review my product. Every time one of my "good guys" asks for something they get what they ask for with out hesitation. The work they put into the review shows and I am sure that at least 50% of the readers know that it is an honest review.

One reason I can do this is because knives on the whole cost a lot less than a complete bike. So if I send a knife to someone and they take their time in reviewing the product it does not cause much, if any financial strain to the bottom line. Where as a bike, especially a flashy new bike that everyone lusts after, is going to set the company back a few dollars. So in order to deal with this the bike companies will set aside a number of bikes, lets say 5, that are for testing/demo. The writers usually only get them for a defined period of time, say a month, because the manufacturer wants that limited supple of demo bikes to go to everyone and get covered in every magazine. So what is the writer to do? They ride it for a couple weeks and get some impressions of the bike and then pass it on. Kind of a no win situation for everyone.

What I would like to see more of is long term tests, say months to a year. That might not sell more bikes initially but in the long run it could prove to be an indication of the possible quality of said manufacturer.

Okay I think I have said enough for now. Back to the knives.

Skinny

[Michael Browne]

Over here at Dirt Rag, we have a strict separation of church and state, I mean, editorial and advertising. This isn't the case with most magazines. I've talked with many editors and writers over my four years at the helm, and I've realized that our minimum six-week testing period is an anomoly. That is, every part we receive must be under a tester's butt for six weeks. If that tester has more than one product or bike to review at one time, he or she must budget their time and make sure that the cumulative period is equivalent to six weeks or more. So that's why our reviews aren't always the most timely.

Some manufacturers don't play with us for this reason. either they "can't afford" to send a bike for that long, or perhaps their attention span has been shortened by the other magazines they work with. Either way, those are the bikes that don't get reviewed by us.

Sometimes I get a call from an advertiser requesting their review to run with their ad. And that's when I tell them we're different entities. Some people respect that--but mainly it's just the readership that appreciates it.

Oh, and that pay-to-play thing? That happens all the time (not with DIrt Rag)... and it's not just bike magazines.

[Tim Grahl]

Pay-to-play... Advertising/review packages... all that makes me sick to my stomach. Just makes it obvious to me that these people have been in the industry to long and have lost sight of what makes of the majority of the riders out there.

I know most people have to save their pennies to buy a new bike and base a lot of their decision on reviews they read. The idea that people should take reviews "with a grain of salt" may be true, but it's trash. Publications need to be held accountable and, IMO, be forthcoming with their review process and how product gets chosen for review.

Which brings us back to blogging and transparency. As people accept typical advertising less and less in exchange for truth and dialogue, it's gonna get harder and harder to hide these underhanded deals. That's my hope for the future anyway.

[Tim Jackson- Masiguy]

Tim, I'm with you man; I hope that things get better. As long as manufacturers need to get product exposure and consumers keep buying magazines that offer "reviews"- no matter how crappy they may be- manufacturers will be inclined to play along. I hope that blogging and other forms of web-based media will continue to have an ever growing presence, since they are so up to date. I don't want to see print mags go away though, as there is something special about going to bed with a good magazine or sitting on the couch in the quiet with a beer.

Bottom line- reviews are good and bad... a necessary evil. Getting the information out to consumers is important as well. I'd prefer to do it face to face at an event, where my passion for my brand can shine, but I can only reach so many people that way.

Tim Jackson
Brand Manager
Masi Bicycles

I know a few of the uk bike journos, and one of the main product testers very well, I've also been roped in for long term testing and feedback on other products.

I can honestly say that everything I've seen them write tallies completley with the feedback given on those products, and I'm not paid in any way for my opinions.

I think the thing forgotten is that an early opinion given after only a few rides by someone who spends all day every day testing stuff and thus having a "feel" for what is right and wrong is worth a lot more than mine - I've only ridden maybe half a dozen different forks, rather than hundreds of them. Equally stuff gets farmed out for long term test with people like me to see how they hold up to regular use - as a 120kg rider I'm a better tester of a clydesdale frame than a 60kg racing snake.

dont know if the US reviews work different, but I know what I've seen is honest and frank opinions on products from experienced reviewers.

[Guy Swarbrick]

This isn't bike insdustry specific. I worked on a reviews-based computer magazine from 1987 to 1992. The same debate raged there. We were accused of taking money to write reviews; we were accused of reviewing products that were advertised more favourably; we were accused of being superficial in the reviews that we ran. Only the last of the three was anywhere near the truth, and there isn't a sensible alternative.

The first accusation is easy to deal with. It never happened. Ever.

The second one is more complex. The way the system worked was this

- the editorial team decided what we wanted to review

- we asked the manufacturers for the products

- we briefed the ad department on what we were reviewing but NEVER what the conclusion of the review was

- they used that information to sell ads but we controlled the magazine layout and ads for a product were NEVER located near the review and the writer of a review wouldn't know the product was being advertised until they saw the finished magazine.

All about as clean as can be. Did we have more trouble getting products to review from some manufacturers than others? Yes. Did that have anything to do with what is we said about them in reviews? Often. Did it have anything to do with whether they advertised with us or what they paid to advertise? I don't know. It was never something they complained about, unlike bad reviews.

The third one is the hardest of all. There were lots of magazines out there and a lot of the 'brand name' manufacturers aren't as big as you might think. They can't afford to give every magazine (and there were even more PC magazines at the time than there are MTB magazines now, even including the online ones), so they have a review stock, which is allocated out for a fixed period of time and then sent somewhere else.

That's certainly going to be the same in the bike industry where the products are as expensive, there are almost as many magazines and the products - especially on the MTB side - are unlikely to be returned in a fit state to be sent out again if they're loaned out for 3 months at a time.

We would have loved to get products for more than a month and we might have been able to do more detailed reviews, but I'm not sure that we would - there were a dozen of us reviewing 60 or 70 products a month, so even if we had them for 6 months I'm not sure how much time we would have been able to spend with them.

Perhaps more importantly, the magazine had about a 3 month lead time from writing an article to it appearing in the shelves. add a month to negotiate with the manufacturer for a review machine and a couple of weeks of testing and you're looking at a review of a 5 month old machine. 6 months was about as long as they were ever on sale for, so by the time you read a review of something it was about to be replaced. A review of a 2005 bike with Tiagra and ally forks on isn't much use when the 2006 bike has Veloce and carbon...

We tried to get around the problem by doing five types of product feature

- Announcements - in the news section. We'd seen one. We knew whether it was total crap, but not much more.

- First looks - we'd had it for between a couple of hours to a couple of days. We hadn't reviewed it, but we had used it and were able to give a useful initial view.

- Reviews - in depth features where the reviewer had used the product extensively for at least a couple of weeks and had carried out some standard tests

- Group tests - Lots of products. Less time with each, but actually using lots of similar products allows you to focus on the differences and get deeper in less time than you can do looking at a product in isolation. These are OK when they're done in house, but almost pointless when they're done by freelance writers who are paid by the word. Ask someone to review 6 products and write 1000 words on each at £150 per 1000 words and you get a couple of hours a product if you're lucky. I hate them; readers love them.

- Long-term tests - Have a product for at least 6 months. Use it every day. Write regular updates on how you're doing. They're the best test of a product, but the least use to most readers as by the time the reports are published the products are no longer available in that form. They're sort of OK as a guide to how good a manufacturer is but, as we all know, the quality of the last product is no guarantee that the next one will be any good.

I've not worked in the bicycle magazine industry, but I see the same sort of things going on as an 'educated' reader and I don't see how things could be much different from a manufacturer's point of view. Review kit is really hard to justify. I don't know what Blue Collar Mountain Bike's readership is, but for a manufacturer to justify sending them a bike which they'll wreck and keep forever they have to be pretty sure that it will sell AT LEAST enough bikes - that they wouldn't have sold anyway - to cover the costs of the review machine. I'd estimate that's about half a dozen, on average. Maybe a couple more.

The problem is that it's impossible to measure. And there are dozens - hundreds, worldwide - of online and paper publications running reviews. And accountants start to notice when you write off a couple of hundred thousand per product launch in review kit that you can't measure the result of. So you have half a dozen review bikes per country, if you're lucky, and you send them out, get them back a couple of weeks later, clean them up and send them out again. And, hopefully, sell them off as ex-demo at the end.

It ain't a perfect system, but it isn't as bad as all that, either.

[Guitar Ted]

Well, it seems to me as I read this that the limitations of print media are perhaps more to blame than the review process itself. Perhaps this is why things like mtbr.com , and blogsites have become increasingly popular. Why should I wait two to three months for an opinion that very well may be biased by business arrangements, when I can surf over to mtbr, and read a "regular joe's" take on the product right away. Not only that, but I can personally contact that product user, ask him specific questions, get them answered, and make my judgements based on the "real world" use of said product. Remember, there are probably, and most likely, several users of that same product, in various places all over the globe to choose from. Of course, this only works as long as there are people willing to jump at the chance to own the "latest and greatest" stuff that comes along. Here, perhaps, is where the print media will have to focus it's energies. They (the print media) have access to the information on new products, where as the consumer obviously does not. I just don't even search out magazine reviews anymore. By the time the review hits the streets in the rags, I can have literally hundreds of opinions from folks running that product on trails just like mine.

[Michael G]

Bike (and other product) reviews and what makes them good or bad has been one of my favorite topics to debate as long as I've been reading them. My beliefs became even stronger after I entered the world of bike mag publishing. I have obsessed over this topic for decades—literally. I won’t bore you with all my thoughts, but since this topic is so near and dear to my heart, I’ll share a little bit of what’s bouncing around in my head.

Much of what has been posted here already is very interesting and basically accurate. Pay-to-play does happen, no doubt about it. For struggling mags it's far too often a must. And for those marketing wonks who have no idea how to handle the media (there have been many in the bike biz), it's one of the few tools they have in their tiny boxes.

The business side of this is interesting, to be sure, but what always concerned me as a consumer and a journalist/editor is the way they are generally predictably written, what information is covered and not covered and how the author's personal relationship with the company or people behind the product affects the story.

As a consumer, I read magazines looking for knowledge, be it in the form of useful advice or information, both of which MUST be new, not regurgitated press releases or rehashed stories from the past.

As an editor familiar with how difficult it is to constantly find new and better ways to present what is basically the same story over and over again, I have always argued (and rarely successfully, I should add) that a bike review should be less of a personal review of a single (or even small "team" of) rider's experience, but more of a story about the product from start to finish. Why not look at it the people behind it, how it came to be, what the designers' goals were and if they succeeded, what niche it fills, who is it designed for, where is it made, how is it made, why is made the way it’s made, etc., etc., etc.?

I could list hundreds of questions that are far more interesting than top tube lengths and tire spec (check the company web site or catalog for this info) or how some 6-inch travel bike handles on a dry, dusty, rocky California loop when I may live in the Mid-West and ride moist shade-covered soil punctuated by tree roots and puddles.

In my mind, that story will always have limited appeal. (For one thing, I’ll only care if I’m looking for a 6-inch travel bike.) But a story that looks at the bike’s background—it’s reason for being, the personalities behind the bike (people make better stories than inanimate objects), the production and marketing machines that brought the bike to market—now that’s good reading that, if well done, will appeal to a far broader audience.

Of course, as many have pointed out to me, there are problems with stories like this. I have been told numerous times that no one wants to sit down and really read. “Our readers just want us to tell them what to buy,” the editor of the world’s most popular bike mag once told me. His comment was later echoed by his replacement and later his replacement, as well as several other staff writers and editors.

I don’t know about you all, but I want to make up my own mind. I’m not looking to be told what to buy. That attitude and approach doesn’t show much respect for the reader.

Another problem with this type of writing is that it actually requires some skill as a journalist. You have to be willing and able to talk directly to lots of people, ask lots of questions, dig, snoop, probe. It’s hard work and it takes skill. And perhaps worst of all, to do it well often means you must sacrifice time in the saddle for your job. (One killer benefit of doing bike reviews is that long rides count as work.)

I used to want a Ritchey mountain bike more than any other, except maybe for a Yeti, a Bontrager, an Ibis or a Fat Chance. And there was a long list of specific parts I wanted, as well. The reason I wanted these bikes and parts was because I had read reviews and bios about the men and companies behind them in an old magazine called Bicycle Guide. The editor then was current VeloNews publisher Ted Costantino. No magazine since has captured my attention like every issue of Bicycle Guide did. What Ted and his phenomenal staff did that no one seems to do now was actually talk to the people behind the products they reviewed and then write what was said in those interviews. I can’t recall ever reading anything like this in the old Bicycle Guide:

“I felt the blood red Italian steed float beneath me as I pedaled up the sun-dappled curves of (insert name here) Canyon. The carbon fiber dream came to life each time I pushed on its pedals. It was torsionally stiff, yet compliant as I raced almost effortlessly over the broken pavement pushed up over the years by the sugar gums that line the roadway. As I finished my first ride on the gleaming scarlet (insert brand name here), I looked down at my computer. To my delight, my heart rate was 20 beats lower than normal and I had knocked more than five minutes off my time daily lunch-time route. And the bike only got better with each new ride.”

While marketing managers pray for this kind of “review,” I hate this drivel. For one thing, it’s as much about the author as it is the bike, and I don’t really give a crap about him or his heart rate or the cool lunch ride he gets to do everyday. Mostly, I hate it because it tells me nothing about the bike or why I should like (or dislike) it.

By telling me about the lives and philosophies of people like Tom Ritchey and Keith Bontrager, I found that I could relate to the products better than if I just looked at a few pretty photos or read some hyperbole-laden garbage about some dude’s ride along the coast. Once I understood the people and the reasons for their decisions, I was more likely to want the product—if those reasons made sense to me, of course.

For example, I remember one article that outlined Tom Ritchey’s obsession with improving bike parts and his desire to make them lighter and easier to use. The article outlined how he took a popular Dia-Compe brake and tweaked it just enough. I thought the man was brilliant and I wanted to buy his stuff. Of course I was a poor semi-employed mountain bike racer at the time so I went for the less expensive Dia-Compes. I never forgot Tom and later, when I had some money, I bought one of his frames and spec’d it out with a bunch of WCS parts and other TR-approved stuff. I still have that bike—the only old-school bike I have managed to hang on to. I did the same thing a few years later with a Bontrager Race Lite. I first read about Keith in Bicycle Guide and later, as a reporter in the bike biz, met and interviewed him. When I was ready to replace my rigid Ritchey with something made for a suspension fork, I went with the Bontrager. I regret selling that bike.

Crikey, I just gave this a look—I’ve been train-of-thought writing until now—and it’s mighty long. I’m not a regular blogger, so I’m not sure what all the protocol for blogging is, but I suspect this it a tad long.

Basically, my premise is this: consistently write well-researched and compelling articles full of new information (I like to call it “news,” which has a dull ring to it, I know.) in order to build a long-term relationship of mutual respect between the editorial staff and the audience on which a sales staff can build a marketing program. (I apologize for that being a bit unrefined, but I just pooped it out so I could put a quick end to this lengthy rant.)

So who am I? I’m the former editor-in-chief of a trade magazine called Bicycle Retailer & Industry News. I worked there for almost 11 years, starting in 1994 as senior editor. Prior to that, I worked for Cycle Press in Tokyo, Japan, as a translator, art director and occasional writer. Since losing my job at BRAIN, I have been looking for work while I attempt to shift career paths mid-life. I’m currently working on a degree in Industrial Design, which is something of an extension of my background as an artist, bike geek, DIYer and 15-plus-year bike industry hanger-on and life-long multi-discipline cyclist.

Again, my apologies for the length, but this seemed like a great place to get a little of this crap off my chest.

[owzgby wvlxays]

rvjkhuz fuqghyor aozynct ntfigx sryu thsqbwi hewozat

[qzghekf rkvmfx]

ajulkrn vedzybwqt ufxy dpmfst jwah dhcobx gptnk http://www.kytwba.hzoa.com

[brcjyzxep mtocrep]

kcuf vbtqpczy wcxrdoqk xpcyrf vlphafi kprjus mcyhljzb [URL=http://www.ptozyiqdw.uaznekcr.com]pcuxnjsm gapnck[/URL]

[Generic Viagra]

in some cases I persive this reviews as a great commercials campaigns, other like Asmtrong campaign are to collect money for a good cause, fight against cancer.

[ported barrels]

We can have a good and fair fight from now on. Thanks for building such effective methods.

[montana divorce]

Im a little surprised with the said information. Very interesting.

[pharmacy]

The first thing that people should know that reviews are something very subjective
.