Let's say you want to conduct a survey in your city, your country, or even worldwide.
Administering it to a group of people who truly represent your population of interest (in terms of demographics and other important factors) will be key to its success, as will your methods of administration: Are you going to cold-call people? Approach them in a mall? Mail out surveys? Each of these methods will skew your data in known ways. What about financial factors? Fact is, most people will expect some form of compensation for their time. How will you pay them?
Fortunately, all these components of survey research can easily be addressed in the same place: highly representative population samples can be accessed nearly immediately, at any time of day and in nearly any country using Amazon's Mechanical Turk. This online service provides access to a global pool of workers who complete tasks for you, such as data organization/cleaning, transcription, and survey-taking.
The graphic below (from the Mechanical Turk website) explains this service in a nutshell. Workers sign up to take surveys, requesters sign up to administer them.
While other excellent services exist online for accessing high quality survey samples (see Schleyer & Forrest, 2000), Mturk, as it's commonly called is simple to use and can be extremely cheap. In fact, with Mturk you decide how much to pay out per survey.
Here I present a step-by-step guide to getting started with Mturk. I'll show how to
- Create an account with Mturk
- Recruit your sample
- Link respondents to your survey
- Download responses
- Approve or reject each response
- Compensate respondents
This looks like a long tutorial, but it is mostly visual references. If you are serious about getting your survey out there, you'll have to start somewhere, and I don't think you'll find a more stream-lined, yet thorough, guide to Mechanical Turk.
While this post specifically addresses academic survey research, these methods are applicable to other types of surveying such as market research.
Introduction: Welcome to Mechanical Turk
Amazon’s Mechanical Turk is becoming a main resource for accessing survey samples in academia. This post is meant to give a basic introduction to using Mturk, and provide a few best practices I have learned. Several resources for more advanced use of Mturk are provided, as are citations to scholarly papers focusing on Mturk.
What is Mechanical Turk?
- Mechanical Turk is an online service that provides access to a global pool of workers who complete tasks for you, such as data organization/cleaning, transcription, and survey-taking
- Created and managed by Amazon.com, it is often referred to as Mturk
- The original mechanical Turk was a chess-playing ‘robot’ made in the 18th century. In reality, this machine was controlled by human intelligence behind the scenes. That's the basic concept of MTurk: artificial-artificial-intelligence. All the boring tasks you wish you had a special robot to complete for you can now be outsourced to the worker pool.
Image: Joseph Racknitz, 1798. The original Mechanical Turk was a chess playing 'robot', which in reality was operated by a person.
- Because the pool of workers on Mturk is so large, you can also use it to access highly representative population samples for survey research.
How Does it Work?
- On the Mturk webpage, requesters post tasks they want completed, called HITS (Human Intelligence Tasks). Workers complete HITS online and are rewarded by the requester
- Everyone involved creates an account with Amazon.com, which acts as arbitrator from respondent recruitment to their compensation
Do Mturk Samples Adequately Represent the Population?
- Online survey samples in general have been criticized as being skewed toward the moneyed, educated, and computer literate (Kaplowitz, Hadlock, & Levine, 2004). However, They have also been praised for their potential global reach; low cost, and the possibility of required completion of the survey for compensation (Brandon, et. al, 2013)
- As for Mturk, studies suggest survey samples accessed through Mturk are more representative than, and show comparable response rates to, traditional respondent pools accessed via phone and mail-in surveys (Evans & Mathur, 2005; Mason & Suri, 2012). Additionally, Mturk can have a far lower price point than traditional survey methods (Brandon, et. Al, 2013)
Creating an Account and Setting Up Your Survey
- To begin using Mturk as a requester, create a requester account at https://requester.mturk.com/begin_signin
- Once you have created your account, begin a new project by clicking the Create tab, and then clicking new project
- This will direct you to the menu below:
- From this menu, your entire survey can be set up and administered within Mturk.
- However, I have found this option less-easy to use, and to have less options than some third party online survey administration services, such as Qualtrics and Google Forms
- I therefore suggest linking to an external survey utilizing the Survey Link option found on the left hand column of the New Project page (if this post is successful, I will probably make a tutorial on using 3rd party online survey software)
- Upon clicking Survey Link, an example of what workers sees upon selecting a Survey Link HIT on Mturk appears
- Click create project- You will be taken to a page where you can name your project, specify who can complete it, and much more.
The next few steps will show you what to do on this page
- Name your project for your own records- Next, enter the information you wish to display to respondents. It is a good idea to make a short, attractive and attention-grabbing title and description, and to use broad keywords
Then, enter:
- Workers’ award per response (consider the hourly wage when deciding, and consider that lower wages will likely impact the quality of your data).
- How many responses you require-
How much time workers have to complete the HIT
Then, enter:
- When the HIT will expire, being removed from the Mturk list of available HITS
- How long workers will have to wait for you to approve their work (covered in later sections)
- At the far right corner of this menu is a link to Advanced survey set-up options
- Clicking this link, a window will appear below. First, click Worker Requirements
- The default worker requirement on Mturk is that they be Master Workers, or workers verified as consentient and accurate at specific types of tasks, such as transcription, photo editing, etc.
- This is probably is too exclusive of criteria if you are trying to work with a random sample of the population - To set up your worker requirements, start by selecting Customize Worker Requirements in the Worker Requirements dropdown menu…
... a new window will appear that allows you to select respondents based on
- Location (nationally or by US state)
- Number of HITS approved
- Overall approval rating by other requesters
- You can also create your own new requirements under the Manage tab, however:
- You cannot test workers for these requirements though the standard requester interface, and have to use Mturk APIs and command line tools (see http://aws.amazon.com/documentation/mturk/)
- Qualification types made by clicking Create Qualification Type in the Manage tab are meant to tag workers with criteria you specify as qualified for later tasks
- After saving your settings, click Design Layout at the very bottom right of the page
- This will take you to a new page (below) showing the default appearance of the HIT Preview workers will see upon selecting your HIT on Mturk
- Type the message you would like workers to see in the text field, and change layout and fonts to your specifications.
- Enter the link to your survey on whatever third party website under Survey Link to direct workers to the HIT
- Note the box for workers to input their completion codes – it is crucial you instruct workers to leave this window open while doing the HIT, and to copy and paste their completion codes into the box so you can verify they took the survey (I will make a later post on a way to generate survey completion codes. Alternatively, many resources exist online instructing on this task)
- Further customization of the Link page is possible via code editing. Click the the Source option to access the html for the page. Images, links, and other features can often be added with a few steps
- For example, adding the Portland State (my alma mater) logo to your page is as easy as pasting the code
into a new section. (See: http://www.w3schools.com/html/html_images.asp for an introduction to changing page appearance with code)
- Once you have changed the appearance of the page and added the appropriate information, click Save and then Preview at the bottom right of the page to see exactly what workers will see (Note the PSU logo that is now in the left corner).
- You will be directed to a new page containing your finished HIT preview
- If you notice something needs to be changed, you can always hit back on your browser and revisit the editing window
- The top bar show what workers will see while browsing through HITS on Mturk. It displays qualification types, the name of the requester (yes, that's my name), the $ amt. per HIT, the duration of the HIT, and the number of HITS available
The lower portion of the preview shows what workers see upon clicking your HIT:
- The summary, survey link, box for completion codes, and the submit button for their final submission of the HIT after entering the completion code
- Click Finish in the lower right boundary of the preview to complete setting up your survey
- Your HIT is now saved under under the Create tab
- From here, you can edit, copy, and delete the HIT- From here you can also Publish your Batch (that is, post your survey on Mturk), the subject of the next section
Publishing Your Batch
- Upon selecting 'Publish Batch', you will be taken to page that reviews your survey settings and worker qualifications…
… and that calculates the cost of publishing your batch, including fees paid to Amazon.com
- This page also shows your projected balance due relative to balance available in your Mturk account
- Click the link below the pricing breakdown to add funds to your account through Amazon Payments
- You will need to set up an Amazon Payments account if you do not already have one
- Once your payment is processed, return to the Publish Batch screen under the Create tab, and click Publish HITS
- You will be taken to a new page showing the Status of your batch (page is accessible though the Manage tab), including current percent completion for your batch, number of assignments completed, average time per assignment, and the average hourly wage you are paying workers (how long the task is taking on average/your compensation)
- You have now published your batch – congratulations! The HITS should now start coming in. You can cancel your batch at any time by clicking the Cancel this batch button
Some things to consider at this point:
- If you find that HITS are being completed at a slow trickle, consider altering your qualifications (is a cutoff of 1000 HITS approved truly needed?). Increasing your compensation might also be necessary
- HITS may also pour in, especially if you are compensating well, which should be considered (if you get all your HITS in an hour, the sample might be biased relative to what time it was in different time zones, for instance)
Approving and Rejecting HITS
- As covered in the Setting Up Your Survey section, an amount of time can be set in days, weeks, etc. for automatic approval (and thus, compensation) of workers’ HITS
- However, you will probably want to review their HITs, checking that completion codes entered match those generated by your third-party survey website
- Mturk allows you to download spreadsheets containing information about each worker who clicked Submit for your HIT. It includes worker worker IDs, and completion codes
- To to access these spreadsheets, open the Manage tab, locate your HIT, and click Results
- The HIT shown below is an older, complete survey (n =600)
- This will take you to page containing information about your (presumably complete) HIT, such as a summary of your qualification types.
- From here, click the Results button (again), found on the center-right of the screen
- You will be taken to a page displaying each completed hit. From here, you have the option to approve/deny your entire batch with one click, and to filter your results by several criteria before downloading the spreadsheet (downloading only the hits you specify)
- Click Download CVS
- You will be taken to a new page containing the download link for your data
- As seen in the image below, you can approve assignments by putting an x in the column marked Approve on the spreadsheet.
- Using Excel, Sort completion codes from the Mturk spreadsheet and from your survey data to compare codes; reject unpaired HITS
- Save the spreadsheet, then upload back to Mturk to complete approvals/rejections in your batch~Some things to consider at this point:
- Of course, matched completion codes are not the only criteria you can can reject for. Simply identify matched completion codes per worker and reject based on whatever criteria you choose
- ALSO - you might consider approving most responses, especially if you are offering a relatively small reward. Rejections tend to come with emails from workers (they can message you through Amazon), and your responsibility to address these comes at a time cost that might be better offset by just compensating the worker. Those few cents per worker do add up, however.
Other Considerations
Attention questions- Because you requesters have relatively minimal interaction with workers, it is good to test whether workers are actually paying attention, and that they are not bots, by adding simple attention questions to your survey such as What is 2 + 4?, or Please confirm you are paying attention by selecting ‘moderately agree’ from the answers below. What to do when workers fail these questions is up to your discretion, but deleting them from your final analysis is probably the most cautious option.
Time lag designsMturk does not have special features for T1-T2 surveys. Having your workers submit their email address so you can contact them for a T2 survey is probably a good idea, or you can decide to have the T2 survey be part of original hit, thus requiring workers return and take the second survey for compensation.
Updating your HITYou can update your settings (such as number of approved HITS to qualify, etc.) at any time. One benefit of doing this is your HIT is sent to the top of the list on Mturk, making it more visible to workers.
Preventing retakes- Dishonest workers taking your survey multiple times is an issue. One way to deal with this is to check worker IDs in the results spreadsheet for duplicates. - A method to prevent retakes within Mturk using a little coding can be found here: http://turkrequesters.blogspot.com/2012/09/academic-requesters-how-to-prevent.html
And that's it!
You now have enough information to access, and get quality data from, a potentially gigantic and highly representative sample of the population. As noted earlier, I did not touch on using third party online survey software that workers actually use to respond the surveys, so stay tuned for that, Steemians.
I hope you enjoyed this post and found it useful. Please let me know about any glaring holes in the instructions, or anything else that seems wrong here.
Other Resources
•Amazon Mechanical Turk Requester User Interface Guide.
–http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/Welcome.html
•Yale guide to administering Qualtrics surveys with Mturk
–http://library.law.yale.edu/news/administering-qualtrics-surveys-mechanical-turk
•Mturk Requester Blog – many tips for requesters and updates on pricing and other issues with Mturk
–http://turkrequesters.blogspot.com
•Regularly updated blog on Mturk demographics
–http://www.behind-the-enemy-lines.com/2015/04/demographics-of-mechanical-turk-now.html
•Summary of Mturk compared to traditional survey methods
–http://priceonomics.com/mechanical-turk-new-face-of-behavioral-science/
•Resource website for workers
–https://www.mturk.com/mturk/help?helpPage=worker
Papers Cited
Brandon, D. M., Long, J. H., Loraas, T. M., Mueller-Phillips, J., & Vansant, B. (2013). Online instrument delivery and participant recruitment services: Emerging opportunities for behavioral accounting research. Behavioral Research in Accounting, 26(1), 1–23.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data?. Perspectives on psychological science, 6(1), 3-5.
Evans, J. R., & Mathur, A. (2005). The value of online surveys. Internet Research, 15(2), 195– 219. doi:10.1108/10662240510590360
Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004). A Comparison of Web and Mail Survey Response Rates. Public Opinion Quarterly, 68(1), 94–101. doi:10.1093/poq/nfh006
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior research methods, 44(1), 1-23.
Schleyer, T. K., & Forrest, J. L. (2000). Methods for the design and administration of web-based surveys. Journal of the American Medical Informatics Association, 7(4), 416-425.