Purpose and Procedures
Mechanical Turk is widely used by businesses and researchers to collect behavioral data from a large number of people in an efficient and cost effective manner. A range of experimental tasks may be posted on Turk, from very simple survey questions lasting under a minute to whole experiments that last for hours.
Participation and Confidentiality
Anyone can sign up for an account on Mechanical Turk as a Requester (i.e. experimenter) or a Provider (i.e. experimental subject). As outlined in the Amazon Mechanical Turk Participation Agreement, by registering to use the site users are certifying that they are at least 18 years of age. Requests for work (“HIT”s) are posted, along with a description and a Requester identification. Each Provider has the opportunity to preview each ‘HIT’, and can choose to participate, and to opt out of participating at any time. If they choose to participate and complete the HIT successfully, they receive payment through the Mechanical Turk interface, which preserves their anonymity.
After Providers submit their answers, the Mechanical Turk interface submits their results to Requesters, along with a unique alphanumeric subject ID. This subject ID allows Requesters to limit the number of times that a particular Provider can participate in related tasks, and to view the results of the same Provider across multiple tasks. Requesters can also limit participantion to Providers with a record of successfully completed HITs.
The Mechanical Turk website does not provide Requesters with any information that would allow them to connect the subject ID to the identity of the original participant. The only personal information about Providers that the Mechanical Turk interface ever provides to Requesters of tasks is the Providers' tax ID information, which is ONLY provided to Requesters if the Providers earn enough in compensation to exceed the IRS tax reporting threshold. Requesters must plan to limit the number of times that Providers may participate in their studies such that they will not exceed this threshold. By doing so, Requesters will not have access to any personal or identifying information about the participants in their studies.
In some cases, MIT-based experimenters may request that participants provide basic demographic information in the HIT (e.g. age, gender, native language).
MIT-based Requesters should include the following disclaimer in the instructions of every HIT:
This HIT is part of a MIT scientific research project. Your decision to complete this HIT is voluntary. There is no way for us to identify you. The only information we will have, in addition to your responses, is the time at which you completed the survey. The results of the research may be presented at scientific meetings or published in scientific journals. Clicking on the 'SUBMIT' button on the bottom of this page indicates that you are at least 18 years of age and agree to complete this HIT voluntarily.
Potential Risks and Discomforts
The major risk to Providers participating in studies through Mechanical Turk occurs when a submitted answer is rejected by the Requester as not adequate. In that case, the Providers incur two costs: they are not paid for that answer, and their overall ‘approval rating’ (i.e. reputation) on the Mechanical Turk is affected. Each Provider on Mechanical Turk has an “approval rating” that is calculated based on how many HITs they have completed, and how many times their reponses have been rejected.
Some criteria for rejections are necessary to ensure and incentivise thoughtful performance. Experience suggests that in the absence of any rejection procedure, data collected on the Mechanical Turk are heavily contaminated by low quality responses.
Amazon.com’s guidelines for the Mechanical Turk explicitly state to Providers that HITs may be controlled for quality, and that it is up to a Requester to accept or reject a HIT, and that if a HIT is rejected, the Provider will not be compensated.
In addition, before deciding to complete an MIT-based HIT, COUHES requires that participants be able to see a preview of the task and instructions, and they must be informed about all requirements and grounds for rejection.
COUHES approves the use of multiple strategies to decide which Providers to reject, in order to ensure quality data. These strategies are designed to detect participants who are not paying sufficient attention to the task and are not answering thoughtfully. It will be up to the individual researcher to decide which strategy to use, but some of the COUHES-approved strategies are outlined below:
- ‘Correct Answer’
Before deciding to complete the HIT, participants will be told "In order to get credit for this HIT, you must read all the questions fully and answer honestly." Requesters will then include a control or catch question. For example, "I have read all the questions fully and answered honestly" or “Move the slider bar all the way to the left if you have read this story carefully.” Alternatively, the question may be a simple objective question about the experiment, such as “To which team were you assigned?” or “what colour was the ball in the movie you just watched?” If a participant answers anything less than “completely agree,” their HIT may be rejected.
- Screening for Double Responders:
Before deciding to complete the HIT, participants will be told “You may only complete this HIT once. If you complete this HIT multiple times, you will be rejected." If a participant completes this HIT multiple times, their subsequent HITs may be rejected.
- Time Requirements:
Before deciding to complete the HIT, Providers will be told “The Requester has placed a time requirement on this HIT. Completing the HIT in too little or too much time will indicate that you have not completed the HIT thoroughly or thoughtfully, or that you have taken a break during completion, and your HIT will be rejected. Once you start the HIT, please eliminate distractions and do not plan to take a break while participating.” If the task involves watching movies, Requesters will create their time limits based on the length of the movies. If the task involves reading, Requesters will create their time limits based on average reading speed and number of words involved in their stimuli. All posted HITs give an upper limit of time allowed to complete the HIT.
- Participant restrictions
In some cases, participants are limited to certain groups (e.g. ‘native English speakers’). If so, the HIT descriptions will clearly define the criteria for inclusion (e.g. ‘You must be a native English speaker to participate”). As part of the task, participants will be asked whether they fit these criteria (e.g. “Are you a native English speaker?”). If they report that they do not fit the criteria, their data may be rejected.
In order to ensure that rejection requirements are fair, Requesters must review their data carefully, and especially review any rejection procedure that is leading to rejections of >20% of HITS. Note that experimenters should be especially careful about rejecting HITs that require a lot of work or time (e.g. more than 5 minutes).
All rejections are made by the Requester after manual inspection of each Provider’s responses. COUHES requests that if an MIT-based Requester does not review a HIT within one week after completion, the HIT should be automatically accepted the Provider should be compensated. Also, Requesters should not use data from rejected (i.e. uncompensated) HITs in any publications.
Potential Benefits
There will be no direct benefits to completing a HIT on Mechanical Turk.
Payment for Participation
Providers will be paid at a rate that is comparable to that of other tasks offered on the Mechanical Turk website and based on the time that Requesters estimate the task will take. Payment is per HIT, and does not depend on the time spent by the Provider, so pay per minute is estimated by the Requester, based on the average time the HIT takes to complete. For instance many tasks currently hosted on the Mechanical Turk pay approximately $0.02 - $0.04 for tasks that take around a minute to complete, and approximately $3.00 - $8.00 for tasks that take 45 – 90 minutes to complete. Payment offered will be adjusted based on how enjoyable or difficult Requesters judge the task to be.
Identification of Investigators
Before participating in a HIT, Providers are able to see the Requester name, which should clearly identify the experimenter as an MIT based researcher, including a specific Lab name. No other identifying information about Requesters is available through the site.
Through the Mechanical Turk interface, Providers are able to email Requesters before and after completing a HIT. It is also common to include a ‘comments’ box in the HIT.
Rights of Research Subjects
Providers are not waiving any legal claims, rights or remedies because of participation in research studies on Mechanical Turk. Providers have access to the Requester ID of every HIT that they complete, and are able to email the Requester through the Mechanical Turk site before or after completing a HIT. If participants should encounter technical errors or have comments/complaints about studies, they will be able to email their complaint to the Requester. COUHES requests that the Requester monitors their email account on a weekly basis, so that emails are responded to within seven days’ time.
Though Amazon.com is not involved in interactions between Requesters and Providers, there are multiple websites relevant to Providers on Mechanical Turk (including http://www.mturkforum.com and http://www.turkers.proboards.com). These sites also provide a space for Providers to read and write comments and complaints regarding Mechanical Turk and specific Requesters.
Updated June 1, 2022