The current reputation system on Mechanical Turk is simply inadequate. The only built-in reputation metrics are the number of completed HITs and the approval rate.
Some people believe that they are adequate as a basic filtering mechanism. They are not.
For example, ask for all workers in your HITs to have 1000 completed HITs and 99% approval rate. You believe that you will only get high quality workers? You are wrong!
I tried to filter workers using just these metrics. I failed. Spammers got me again. (And once in, spammers submit a lot of crap. It costs nothing.) I questioned why. How can it be? And then I realized: It is trivial to beat these metrics.
Let's see the effort it takes to beat the system.
The mission: Become a top Turker, 100% approval rate and 1000 completed HITs.
- Step 1: Login as a requester. Post a task, with 1000 HITs. Each HIT pays 1 cent. Total cost: \$15. Out of these, \$10 go to the worker, \$5 go to Amazon. The title of the HIT: "Write a 500 word review". No sane worker will touch these HITs. Done. Logout.
- Step 2: Login as worker, using a different email. Complete and submit the 1000 HITs created in Step 1. You just need to click submit 1000 times. Bored? iMacro and Greasemonkey can help. Done. Logout.
- Step 3: Login as a requester again. Approve all submitted HITs. Pay the \$15. Amazon gets \$5. The worker account has the remaining \$10. Done. Logout.
Your worker account is a top Turker now. 1000 completed HITs and 100% approval rate. Congratulations! You have a license to spam.