Monday, November 18, 2019

Mechanical Turk, 97 cents per hour, and common reporting biases

The New York Times has an article about Mechanical Turk in today's print edition: "I Found Work on an Amazon Website. I Made 97 Cents an Hour". (You will find a couple of quotes from yours truly).

The content of the article follows the current zeitgeist: Tech companies exploiting gig workers. 

While it is hard to argue that there are tasks on MTurk that are really bad, I think that the article paints an unfairly gloomy picture of the overall platform.

Here are a few of the issues:
  • Availability and survivorship bias. While the paper does describe accurately the cesspool of low-paying tasks that are available on Mechanical Turk, it fails to convey the fact that these tasks are available on the platform because nobody wants to work on them. The tasks that are easily available for everyone are the ones for which nobody competes to grab: low-paying, badly designed tasks.
  • The activity levels of workers follow a power-law. We have plenty of evidence that a significant part of the work on MTurk is done by a small minority of workers. While it is hard to have a truly accurate measurement of what percent of the workers do what percent of the tasks, the 1% rule is a good approximation. For example, in my demographic surveys, where I explicitly limit the participation to only once per month, 50% of the responses come from 5% of the participants. Expect the bias to be much stronger in other, more desirable tasks. Such a heavily biased propensity to participate introduces strong sampling problems when trying to find the right set of workers to interview.
  • Doing piecemeal work while untrained results in low pay. This is a pet peeve of mine, for all the articles of the type "I tried working on MTurk / driving Uber / delivering packages / etc / and I got a lousy pay". Well, if you work piecemeal on any task, the tasks will take a very long time initially, and the hourly wage will suck. This will hold for Turking, coding, lawyering, or anything else. If someone decides to become a freelance journalist, the first few articles will result in abysmally bad hourly wages as well; expert freelance writers often charge 10x the rates that beginner freelancer writers charge, if not more. I am 100% confident that the same applies to MTurk workers as well: Experienced workers make 10x what beginners make.
Having said that, I do agree that Amazon could prohibit tasks that are obviously paying very little (as a rule of thumb, it is impossible to get paid more than minimum wage when the HIT is paying less than 5c/task). But I also think that regular workers are smart enough to know that and avoid such tasks.