People that have experience with Mechanical Turk know that getting long tasks done on Mechanical Turk is tricky. While it is relatively easy to get small tasks done quickly, it is much more difficult to estimate how long a big task will take. The "estimated time" given by the Mechanical Turk interface is really crappy and provides pretty much no guidance if you expect your task to last longer than a day.
Naturally questions like this arise: When is it best to post a task? How can I minimize my waiting time?
Trying to understand better how tasks are being completed on Mechanical Turk, I started crawling Mechanical Turk every few minutes collecting data about the HITs, the requesters, how long each HIT is available and so on.
Queue
The first outcome of this effort was the Mechanical Turk Monitor, a visualization tool that shows how many projects are available at any given time, how many HITs, and the available rewards (see the old post).
Arrival process and Serving process
This tool was effectively showing the size of the "queue". However, it did not reveal neither how many tasks arrive per day on MTurk, nor how much work gets done on MTurk every day. So, last week I decided to display this information, and show the activity of the requesters and the corresponding activity of Turkers every day:
- Activity of requestors: How many tasks are posted per day
- Activity of Turkers: How many tasks are completed per day
Posting Activity
Now, we can start scratching the surface on how things get done on Mechanical Turk. A first pass is to see the statistics for what is being posted over time:
The x-axis depicts time, and the y-axis is the value of the HITs being posted every day. The blue line depicts the total value of the HITs being posted. One immediate observation is that there is some significant periodicity. Taking the 7-day average (red line) smooths significantly the curve. This indicates that there is some strong weekly periodicity.
Let's take a look at the distribution of posting activity over the days of the week:
The plot shows the distribution of the activity for every day of the week. By activity, we define the total value of HITs being posted on each day. As we can see, weekends tend to be significantly more quiet than weekdays. In fact, even Mondays tend to be relatively quiet, perhaps because requesters prepare their HITs that are then being posted on Tuesdays :-)
Well, the plot is not very surprising. Lots of activity during the workdays, less activity over the weekends.
Workers Activity
The interesting result though comes when we look at the activity of the Turkers:
It seems that Turkers are not in sync with the requesters. In fact, the activity on Saturdays us comparable to the activity during the weekdays. Surprisingly, Mondays tend to see significantly less activity. (Perhaps due to the small number of tasks being posted over the weekend?).
Conclusion
What is clear is that there is a relative lag between the activity of requesters and workers. Although it is hard to figure out causality from these figures, it seems that Fridays and Saturdays are good days to post tasks on Mechanical Turk. Relatively low competition for the attention of workers, and significant level of workers activity during Saturday.
So, now you know: Post your HITs on Friday and go away...
