Wednesday, September 26, 2012

Do not threaten workers in your instructions

This was the title of a thread on TurkerNation. The screen cap pretty much says it all. This turns off workers.

Post from turkernation used with permission

 Let me explain a little further so you can understand why this is a pretty big deal for workers.

Rejections- All workers get them eventually.  Workers understand that sometimes this happens, but they also know that any rejection will impact their approval percentage. Approval percentage is one of the most widely used qualifications that requesters use to find qualified workers. The best and most experienced workers do their best to keep this number above 99%.
For a new worker on Mturk, a few rejections can impact their approval percentage greatly. For every one rejection, a worker has to have 100 approvals to get their approval percentage back above 99%.
Most experienced turkers are in the 99.5-99.9% approval range, so a single rejection means these workers will have to complete 500-1000  HITs to remove the effect of one rejection.
This is the workforce that you want working on your HITs. They have the highest accuracy and have usually completed a wide variety of different HITs that give them the experience you need to work on any task you have in mind. If you turn them off in the first few sentences of your instructions, you will not be pleased with your results.

Blocks- Blocks are a big no no. ONLY use blocks when you are positive that a worker was blatantly trying to cheat or steal from you. Amazon does not go out of their way to tell requesters how big of a deal blocks are. When a worker is blocked, this can trigger Amazon to do a review of a workers account.
This is the sequence of events when this happens -
Day 1 - Amazon receives notice of block on worker account
Day 2 - Worker account is automatically suspended (cannot work or earn money)
Day 3-7 or more - Amazon reviews account.
Week or two later - Amazon makes a decision to permanently ban worker or reinstate account.

This is possibly a week or two of not being able to work or earn a living. So the instructions state "Note: we will review all work and reject any work that we find unsatisfactory. Workers whose work is rejected will also be blocked" Although that is what is written, the worker sees this, "we could block you and you might lose your Mturk account."

I am not saying that a requester should pay for work that is not done properly. If you have clear instructions, have a well designed HIT, and a worker does not submit usable work, you are within your rights to reject. BUT, you are better off forming some sort of communication with your workers and publishing in small batches with your qualified workforce prior to rejecting dozens of HITs. If you are going to block, use blocks to rid Mturk of the scamming workers who try to undermine the workplace and not to punish honest workers who may have made a mistake.

Below is the worker's view of how this should be handled....
Used with permission of author

1 comment:

  1. Last year I'd done a handful of HiTs for a requester. the first three were approved. the next two were rejected. when I politely mentioned to the requester that the rejected two were just as good as the other three, could he please tell me where I went wrong? He BLOCKED me. I wasn't the only one: several people on turkopticon mentioned the same thing.

    Requesters, please don't block people for politely asking how they could have done better.