What Tasks Are Posted on Mechanical Turk?

A few months back, I got an invitation from Michael Bernstein (of Soylent fame) to write a small article about Mechanical Turk for the student magazine of ACM, the ACM XRDS (aka Crossroads). I could have written a summary of past research, a position paper, or anything that I find interesting.

Instead of summarizing and resubmitting already published material, I decided to push myself and start analyzing some data that I have been collecting about the Mechanical Turk marketplace. In the past, I analyzed the data about the demographics of the workers on Mechanical Turk. However, we have limited analysis for the requester side of the market, and for the type of tasks being posted.

My goal was to put some very preliminary analysis in place, just to scratch the surface of a variety of a questions that I have heard over time. Hopefully this will push me to start working more towards getting the answers, and will inspire some interesting new questions for the students and others that read the article. A preprint of the paper is available through the NYU Faculty Digital Archive and the print version should appear sometime early in 2011.

Data Set

I have been collecting data about the marketplace through my Mechanical Turk Tracker. The tracker collects complete snapshots of the marketplace, every hour, starting from January 2009. For the analysis, I took the data from the period of January 2009 till April 2010: The snapshots has a total of:

• 165,368 HIT groups
• 6,701,406 HITs
• 9,436 requesters
• $529,259 rewards

These numbers, of course, do not account for the redundancy of the posted HITs, or for HITs that were posted and disappeared between our hourly crawls. Nevertheless, they should be good approximations (within an order of magnitude) of the activity of the marketplace.

Top Requesters

The first question that I looked at was an analysis of the tasks that are being posted on Mechanical Turk. One way to understand what types of tasks are being completed in the marketplace is to find the “top” requesters and analyze the HITs that they post. By ranking requesters according to the sum of the posted rewards, we get the following list, showing the level of activity and the type of tasks that these requesters post. (Note: To avoid skewing the data towards one-shot requesters, I excluded from the list a requesters that were active only for small periods of time or requesters that posted only a small number of HITs. The goal was to find not only the requesters that post big tasks, but also requesters that do that consistently over time.)

So, transcription, classification, and content generation seems to be a common activity on Mechanical Turk. This indicates that people have developed sufficient best practices and can actually get quality work done. (If not, they would not be posting so many tasks.)

Top Keywords

We also wanted to get a feeling of the tasks that are being posted in the market, across all requesters. The table below shows the top-50 most frequent HIT keywords in the dataset, ranked by total reward amount, # of HITgroups, and # of HITs.

Beyond the tasks identified before, we also see data collection, image tagging, website feedback, and usability tests to be common tasks being posted in the marketplace.

In future posts, I will post further analysis of other aspects of the AMT marketplace.