Wednesday, July 30, 2008

Mechanical Turk Allows Bulk Submissions Using Templates

Mechanical Turk is a very useful service that I have used repeatedly in the past for a variety of tasks. While it was a great tool, one of the shortcomings was the lack of a web-based interface support to create batches of tasks. Using the web it was only possible to create a single task at a time, potentially asking many Turkers to complete it.

If someone wanted to generate a large number of similar tasks (e.g., submit 1000 query-document pairs for relevance judgments as opposed to 1 query-document pair) then the only option was to use some command-line tools, or use the MTurk API. Admittedly, the tools were easy to use, but still the need to resort to "programming" was a problem for many people with no programming background.

So, today Amazon released the ability to generate "templates" that will automate the generation of such tasks. The basic idea: You generate a template, and specify in the template a variable placeholder (e.g., #document). Then you upload a separate file with the data that will be used to populate the placeholder, and you are done. A set of HITs are then generated automatically.

Amazon has already created some basic HIT templates that illustrate the process, showing example tasks such as "Get Product Name from Image," "Are these products the same," "Evaluate the Effectiveness of Search Results" and so on. The Getting Started Guide illustrates how easy is to generate the batches of HITs.

Nice! I may finally generate some batch HITs myself, instead of asking my students to do all the work for me :-)