Sunday, October 14, 2012

Using oDesk for microtasks

Quite a few people keep asking me about Mechanical Turk. Truth be told, I have not used MTurk for my own work for quite some time. Instead I use oDesk to get workers for my tasks, and, increasingly, for my microtasks as well.

When I mention that people can use oDesk for micro-tasks, people get often surprised: "oDesk cannot be used through an API, it is designed for human interaction, right?" Oh well, yes and no. Yes, most jobs require some form of interviewing, but there are certainly jobs where you do not need to manually interview a worker before engaging them. In fact, with most crowdsourcing jobs having both the training and the evaluation component built in the working process, the manual interview is often not needed.

For such crowdsourcing-style jobs, you can use the oDesk API to automate the hiring of workers to work on your tasks. You can find the API at (Saying that the API page is, ahem, badly designed, is an understatement. Nevertheless, it is possible to figure out how to use it, relatively quickly, so let's move on.)

Here are the typical steps for a crowdsourcing-style contract on oDesk:
  • First, post a job: Use the "Post a Job" call from the Jobs API
  • Once the job is posted, poll the job openings to find who applied: Use the "List all the offers" call from the Offers API
  • Examine the details of the contractors that bid on the job: Use the "Get Offer" from the Offers API, to examine the details of each contractor. For example, for a task we had to have at most 10 people from a given country. So, the first 10 people from each country were hired, while subsequent applications from a country that had already 10 applicants were denied. Other people may decide not to hire contractors with less than 50 hours of prior work. It seems to be an interesting research topic to intelligently decide what aspects of the contractor matter most for a job, and hire/decline applications based on such info.
  • Make offers to the contractors: [That is the stupid part: Apparently the API does not allow the buyer to simply "accept" the bid by the contractor, although this is trivially possible through the web interface]. Use the "Post a Job" call, and create a new job opening for the contractor. Then use the "Make Offer" call from the Offers API, to generate an offer for the contractor(s) that you want to hire.
    • If you do not want to pay per hour, but rather per task, create an hourly contract, but set the maximum working hours per week at zero. Yes, this is not a mistake. You will be using the Custom Payments functionality to effectively submit "bonus payments" to the contractor.
    • Typically, it is better to have a mixture of both hourly wage and a fixed price component. You can have a no-hourly-wage policy by setting at 0 the maximum hours that can be charged, simulating MTurk. Or you can specify the hourly wage, and set the limit of how many hours can be charged per week.
  • Direct the contractor how to work: For that use the Message Center API, to send a message to the contractor, with the URL where you host your task. [Note: oDesk does not provide functionality for handling the task execution, so it is up to you to build that infrastructure. If you have ever built an "external HIT" on MTurk, you are ready to go. Just now you need to send the oDesk workers a url, where they can login to your website, and their username/password. You can go full force and allow an oDesk authentication, but this seems a little bit too much for me.]
  • Whenever the contractor has completed enough tasks, use the Custom Payment API to submit the payment. Repeat as needed.
  • When the task is done, end the contract using the contracts API.
That's all folks! In the next few weeks, I will try to post the code for some of the crowdsourcing experiments that we conducted with oDesk.