Budget-Optimal Crowdsourcing Using Low-Rank Matrix Approximations
Name
Karger_Budget-optimal crowdsourcing.pdf
Size
259.33 KB
Format
Adobe PDF
Checksum (MD5)
bc7d219040bba90bcfb93fd3025bc87c
Author(s)
Karger, David R.
Oh, Sewoong
Shah, Devavrat
Date Issued
September 2011
Journal
Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2011
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Karger, David R., Sewoong Oh, and Devavrat Shah. “Budget-optimal Crowdsourcing Using Low-rank Matrix Approximations.” 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2011. 284–291.
Version
Author's final manuscript
Abstract
Crowdsourcing systems, in which numerous tasks are electronically distributed to numerous "information piece- workers", have emerged as an effective paradigm for human- powered solving of large scale problems in domains such as image classification, data entry, optical character recognition, recommendation, and proofreading. Because these low-paid workers can be unreliable, nearly all crowdsourcers must devise schemes to increase confidence in their answers, typically by assigning each task multiple times and combining the answers in some way such as majority voting. In this paper, we consider a model of such crowdsourcing tasks and pose the problem of minimizing the total price (i.e., number of task assignments) that must be paid to achieve a target overall reliability. We give a new algorithm for deciding which tasks to assign to which workers and for inferring correct answers from the workers' answers. We show that our algorithm, based on low-rank matrix approximation, significantly outperforms majority voting and, in fact, is order-optimal through comparison to an oracle that knows the reliability of every worker.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike 3.0
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/Allerton.2011.6120180
