Preventing retakes on your survey is the most common issue that requesters have with mTurk. This guide explains an easy way to prevent retakes just by using the Amazon mTurk Command Line Tools (CLT) that work on Windows, OS X and GNU/Linux.
Note about Amazon's Block MethodA Google search will yield this forum post where an mTurk representative tells requesters to use the block worker function to prevent survey retakes. Unfortunately, Amazon's system is not perfect, so even if you do this a worker may still get the following form email:
Greetings from Mechanical Turk.Not only that, if you want to give our multiple surveys of the same type workers who took them before won't be able to take a different kind of survey from you.
We regret to inform you that you were blocked from working on HITs by the following requester(s):
Requesters typically block Workers who submit poor quality work.
Requesters rely on Mechanical Turk for high quality work results. In order to maintain quality, we continuously monitor service activity. Should additional Requesters block you in the future, we may suspend your account. Please ensure your work quality is at a high standard. We encourage you to read the HIT instructions and follow them carefully.
We realize that this block may be a onetime occurrence for you. Should you maintain high work quality with no further complaints for the next few months we will dismiss this event.
The Mechanical Turk Team
IsaacM says this should never happen but experience begs to differ as requesters who use this method still get these emails sent out to workers. (There are many posts on the mTurk worker forum to this effect.) If there is a "glitch" in mTurk workers are going to be emailing you pleading for an unblock and you will get a bad reputation which may make responses from future surveys come more slowly or not at all.
Preferred Method: QualificationsInstead of potentially causing a lot of trouble, you can use mTurk's qualifications to keep workers out of surveys. Basically, how it works is this: Your survey has a qualification pre-attached to it called "Did my survey" and it requires a value of 0 from a worker, a worker requests the qualification and it is auto-granted to them at a value of 0, and once they take the survey the qualification is incremented by 1.
First, create a file that looks like this, and name it something like
To make the qualification, execute this command with the mTurk Command Line Tools (note: In this post, I've used the Windows syntax for all command line examples. For OSX/Linux, prepend the characters
name:No retakes please! description:Prevent retakes on my survey keywords:prevent, retakes autogranted:true autograntedvalue:0
./to the beginning of the command and append
.shto the end of the first word, e.g.
You will get a QualTypeID printed to standard output as well as to a file called
createQualificationType -properties noretakes.properties
noretakes.success. You need this ID to change the values later.
Make sure to add this qualification to your
hit.propertiesfile if you are making a new HIT with the CLT or to the necessary qualifications in the Hosted Requester GUI. Remember that the value should be
Once the first run of the survey is complete, you can now raise everyone's values.
Whether you used the
getResultscommand line tool or the web UI to get your results file, you should still have all of the work ID's who submitted work to your survey. Create a tab-delimited text file with the columns
score. A few programs can create these files, such as Microsoft Excel and LibreOffice Calc. In Excel, they are called
.tsv (tab separated values)files while in Calc they are saved as
.csvbut with a different delimiter (namely,
Your file should look something like this, with the symbol → representing a tab:
Then, run the following command to update everyone's qual score (note, the
workid→score A1EXAMPLE→1 A2EXAMPLE→1 A3EXAMPLE→1
-qualtypeidparameter takes the QualTypeID generated earlier and stored in
updateQualificationScore -qualtypeid TPREVENTRETAKESQUALIDEXAMPLE -input noretakes.tsv
Harder method: Internal listsA harder method to prevent retakes is to ask for a worker's work ID when they begin your survey, and if it's found on a list, tell them to return the HIT. Doing that is far out of the scope of this article, though, and this method is better as it will prevent any workers from even accepting a survey they cannot do, which will leave them on the site longer for people who can do them!
I hope this was useful for you and I hope your survey gets lots of replies. :) Remember to pay workers fairly! (at least 12 cents per minute)