Analytic Methods for Optimizing Realtime Crowdsourcing
Name
ci2012-karger-1204.2995.pdf
Size
508.29 KB
Format
Adobe PDF
Checksum (MD5)
5e633ad8637a105bd8a3754b31b89b8f
Author(s)
Brandt, Joel
Karger, David R
Bernstein, Michael Scott
Miller, Robert C
Date Issued
April 2012
Journal
Proceedings of the Collective Intelligence Conference, 2012
Citation
Bernstein, Michael S. et al. "Analytic Methods for Optimizing Realtime Crowdsourcing." Collective Intelligence Conference 2012, Cambridge, MA, USA, 18-20 April, 2012.
Version
Author's final manuscript
Abstract
Realtime crowdsourcing research has demonstrated that it is possible to recruit paid crowds within seconds by managing a small, fast-reacting worker pool. Realtime crowds enable crowd-powered systems that respond at interactive speeds: for example, cameras, robots and instant opinion polls. So far, these techniques have mainly been proof-of-concept prototypes: research has not yet attempted to understand how they might work at large scale or optimize their cost/performance trade-offs. In this paper, we use queueing theory to analyze the retainer model for realtime crowdsourcing, in particular its expected wait time and cost to requesters. We provide an algorithm that allows requesters to minimize their cost subject to performance requirements. We then propose and analyze three techniques to improve performance: push notifications, shared retainer pools, and precruitment, which involves recalling retainer workers before a task actually arrives. An experimental validation finds that precruited workers begin a task 500 milliseconds after it is posted, delivering results below the one-second cognitive threshold for an end-user to stay in flow.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.48550/arXiv.1204.2995
