Mechanical Turk, Low Wages, and the Market for Lemons

In HCOMP this year, one of the memorable and discussed presentations (although highly unconventional) was by M. Six Silberman who discussed the "Sellers' problems in human computation markets". The basic question: can we protect the workers there from exploitation and from sweatshop salaries?

Luis von Ahn posted a similar post on his blog. In the comments of the blog post, someone suggested that the low wages on Mechanical Turk is simply the result of high supply of workers and low demand for their work. As there is more supply, the salaries drop. And having minimum wages, would interfere with the free market.

I actually disagree with this interpretation. First of all, there is no oversupply of labor on Mechanical Turk. The distribution of completion times (follows a power law), suggests that the market operates at maximum capacity. My gut instinct actually tells me that there are not enough workers available for the posted work, not vice versa.

I can hear the protests: If there is not enough supply of workers, why don't requesters simply increase the offered prices?

My explanation: The requesters already pay minimum wages for work that is worth minimum wage. How is that possible given the effective hourly rate of \$2/hour?

The basic problem: Spammers. Given that many large tasks attract spammers, most requesters rely on redundancy to ensure quality. So instead of having a single worker to do a task, they get 5 workers to work on it. This increases the effective rate from \$2/hr to \$10/hr.

Effectively, what Amazon Mechanical Turk is today is a market for lemons, following the terminology of Akerlof's famous paper, for which he got the 2001 Nobel prize.

A market for lemons is a market where the sellers cannot evaluate beforehand the quality of the goods that they are buying. So, if you have two types of products (say good workers and low quality workers) and cannot tell who is whom, the price that the buyer is willing to pay will be proportional to the average quality of the worker. So the offered price will be between the price of a good worker and a low quality worker. What a good worker would do? Given that good workers will not get enough payment for their true quality, they leave the market. This leads the buyer to lower the price even more towards the price for low quality workers. At the end, we only have low quality workers in the market (or workers willing to work for similar wages) and the offered price reflects that.

This is exactly what is happening on Mechanical Turk today. Requesters pay everyone as if they are low quality workers, assuming that extra quality assurance techniques will be required on top of Mechanical Turk.

So, how can someone resolve such issues? The basic solution is the concept of signalling. Good workers need a method to signal to the buyer their higher quality. In this way, they can differentiate themselves from low quality workers. Unfortunately, Amazon has not implemented a good reputation mechanism. The "number of HITs worked" and the "acceptance percentage" are simply not sufficient signalling mechanisms.

Here are some ideas:

• Allowing workers to get endorsements from reputable requesters (to avoid scam rings like on eBay)
• Allowing requesters to post machine readable feedback on the performance of the workers, disconnecting evaluation from the approval rate.
• Certifications and qualification tests that indeed measure ability on different tasks (e.g., language abilities, reading comprehension tests, etc)
• Publishing the reputation history of the workers, so that requesters can evaluate the quality of the worker.

Of course, similar measures can be adopted for requesters! There is a symmetric market for lemons on that side! Scam requesters post HITs, behave badly, and cause good workers to avoid any newcomer. New requesters then get only low quality workers, get disappointed with the quality of the results and they leave the market.

In other words, Amazon can only gain by taking the time to build a more robust reputation system on top of Mechanical Turk. Trust is at the very core of marketplaces. If Mechanical Turk wants to "grow up", then a good reputation system for both sides of the market is grossly overdue.