Thursday, January 22, 2009

Soliciting Reviews on Mechanical Turk

A few days ago, Mechanical Turk was featured in a not-so-flattering story: Mike Baynard, a Business Development Representative working for Belkin has posted HITs on Mechanical Turk asking Turkers to write 5 star reviews on Amazon, for a set of Belkin products that were getting mainly negative reviews. Furthermore, Mike Baynard was asking Turkers, after posting the 5/5 review, to vote as "not helpful" the negative reviews that appeared on Amazon. The story was picked by major tech sites (Gizmodo, Slashdot) and, fortunately, users put the blame on Belkin and not on Mechanical Turk.

However, this got me thinking. How often is Mechanical Turk used for such sort of activities? Fortunately, a few weeks back I set up a crawler that visits Mechanical Turk periodically and keeps track of the tasks posted there.

So, in my first use of this MTurk archive, I digged in and tried to find review-related HITs. I discovered about 100 HIT groups posted over the last couple of weeks. Except for three HITs posted by Michael Bayard, the rest did not seem to explicitly solicit positive reviews. In fact, most of the requests seem to be legitimate and in my opinion, ethical. Yes, in an ideal world all websites would get millions of users submitting reviews and generating network effects, but sometimes it is better to pay and have a review than not having a review at all!

(The reviews are listed in a table, which is contained within an IFRAME; this means that you will need to visit the blog page to see it as RSS readers typically do not render IFRAMEs)

Now, the question is: Are these solicited/paid reviews better or worse than the reviews posted by users without any financial incentive? I expect to have some results in that front rather soon.