dc.contributor.author | Karger, David R. | |
dc.contributor.author | Oh, Sewoong | |
dc.contributor.author | Shah, Devavrat | |
dc.date.accessioned | 2012-09-28T13:01:42Z | |
dc.date.available | 2012-09-28T13:01:42Z | |
dc.date.issued | 2011-12 | |
dc.identifier.uri | http://hdl.handle.net/1721.1/73460 | |
dc.description.abstract | Crowdsourcing systems, in which tasks are electronically distributed to numerous
“information piece-workers”, have emerged as an effective paradigm for humanpowered
solving of large scale problems in domains such as image classification,
data entry, optical character recognition, recommendation, and proofreading. Because
these low-paid workers can be unreliable, nearly all crowdsourcers must
devise schemes to increase confidence in their answers, typically by assigning
each task multiple times and combining the answers in some way such as majority
voting. In this paper, we consider a general model of such crowdsourcing
tasks, and pose the problem of minimizing the total price (i.e., number of task assignments)
that must be paid to achieve a target overall reliability. We give a new
algorithm for deciding which tasks to assign to which workers and for inferring
correct answers from the workers’ answers. We show that our algorithm significantly
outperforms majority voting and, in fact, is asymptotically optimal through
comparison to an oracle that knows the reliability of every worker. | en_US |
dc.language.iso | en_US | |
dc.publisher | Neural Information Processing Systems | en_US |
dc.relation.isversionof | http://nips.cc/Conferences/2011/Program/accepted-papers.php | en_US |
dc.rights | Creative Commons Attribution-Noncommercial-Share Alike 3.0 | en_US |
dc.rights.uri | http://creativecommons.org/licenses/by-nc-sa/3.0/ | en_US |
dc.source | MIT web domain | en_US |
dc.title | Iterative Learning for Reliable Crowdsourcing Systems | en_US |
dc.type | Article | en_US |
dc.identifier.citation | David R. Karger, Sewoong Oh, Devavrat Shah. "Iterative Learning for Reliable Crowdsourcing Systems" Neural Information Processing Systems, 2011: 1953-1961. | en_US |
dc.contributor.department | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science | en_US |
dc.contributor.mitauthor | Karger, David R. | |
dc.contributor.mitauthor | Oh, Sewoong | |
dc.contributor.mitauthor | Shah, Devavrat | |
dc.relation.journal | Advances in Neural Information Processing Systems 24 | en_US |
dc.eprint.version | Author's final manuscript | en_US |
dc.type.uri | http://purl.org/eprint/type/ConferencePaper | en_US |
dc.identifier.orcid | https://orcid.org/0000-0003-0737-3259 | |
dc.identifier.orcid | https://orcid.org/0000-0002-0024-5847 | |
mit.license | OPEN_ACCESS_POLICY | en_US |
mit.metadata.status | Complete | |