Using mechanical turk to obtain and analyze English acceptability judgments
Name
Gibson, Piantadosi & Fedorenko, April 2011[1].pdf
Size
405.23 KB
Format
Adobe PDF
Checksum (MD5)
9b42f145496421458600b475e42c8f22
Author(s)
Gibson, Edward A.
Piantadosi, Steven Thomas
Fedorenko, Kristina
Date Issued
August 2011
Journal
Language and Linguistics Compass
Publisher
John Wiley & Sons, Inc.
Citation
Gibson, Edward, Steve Piantadosi, and Kristina Fedorenko. “Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments.” Language and Linguistics Compass 5.8 (2011): 509–524. Web.
Version
Author's final manuscript
Abstract
The prevalent method in theoretical syntax and semantics research involves obtaining a judgment of the acceptability of a sentence/meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. The weakness of the traditional non-quantitative single-sentence/single-participant methodology, along with the existence of cognitive and social biases, has the unwanted effect that claims in the syntax and semantics literature cannot be trusted. Even if most of the judgments in an arbitrary syntax/semantics paper can be substantiated with rigorous quantitative experiments, the existence of a small set of judgments that do not conform to the authors’ intuitions can have a large effect on the potential theories. Whereas it is clearly desirable to quantitatively evaluate all syntactic and semantic hypotheses, it has been time-consuming in the past to find a large pool of naïve experimental participants for behavioral experiments. The advent of Amazon.com’s Mechanical Turk now makes this process very simple. Mechanical Turk is a marketplace interface that can be used for collecting behavioral data over the internet quickly and inexpensively. The cost of using an interface like Mechanical Turk is minimal, and the time that it takes for the results to be returned is very short. Many linguistic surveys can be completed within a day, at a cost of less than $50. In this paper, we provide detailed instructions for how to use our freely available software in order to (a) post-linguistic acceptability surveys to Mechanical Turk; and (b) extract and analyze the resulting data.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike 3.0
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1111/j.1749-818X.2011.00295.x
