MIT Libraries logoDSpace@MIT

MIT
View Item 
  • DSpace@MIT Home
  • MIT Open Access Articles
  • MIT Open Access Articles
  • View Item
  • DSpace@MIT Home
  • MIT Open Access Articles
  • MIT Open Access Articles
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

A Generic Human–Machine Annotation Framework Based on Dynamic Cooperative Learning

Author(s)
Zhang, Yue
Thumbnail
DownloadZhang19-AGH_Cybernetics.pdf (1.546Mb)
Open Access Policy

Open Access Policy

Creative Commons Attribution-Noncommercial-Share Alike

Terms of use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Metadata
Show full item record
Abstract
The task of obtaining meaningful annotations is atedious work, incurring considerable costs and time consumption.Dynamic active learning and cooperative learning are recentlyproposed approaches to reducing human effort of annotatingdata with subjective phenomena. In this work, we introduce anovel generic annotation framework, with the aim to achieve theoptimal trade-off between label reliability and cost reduction bymaking efficient use of human and machine work force. To thisend, we use dropout to assess model uncertainty and therebyto decide which instances can be automatically labelled by themachine and which ones require human inspection. Additionally,we propose an early stopping criterion based on inter-rateragreement in order to focus human resources on those ambiguousinstances that are difficult to label. In contrast to the existingalgorithms, the new confidence measures are not only applicableto binary classification tasks, but also regression problems. Theproposed method is evaluated on the benchmark datasets for non-native English prosody estimation, provided in the INTERSPEECHComputational Paralinguistics Challenge. In the result, the noveldynamic cooperative learning algorithm yields .424 Spearman’scorrelation coefficient compared to .413 with passive learning,while reducing the amount of human annotations by 74 %.Index Terms—Human-Machine Systems, Active Learning,Semi-supervised Learning, Confidence Measures, Inter-raterAgreement
Date issued
2020-03
URI
https://hdl.handle.net/1721.1/124308
Department
Massachusetts Institute of Technology. Media Laboratory
Journal
IEEE Transactions on Cybernetics
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Zhang, Yue, Andrea Michi, Johannes Wagner, Elisabeth Andŕe, Bj̈̈orn Schuller, Felix Weninger. "A Generic Human–Machine Annotation Framework Based on Dynamic Cooperative Learning." IEEE Transactions on Cybernetics 50 (2020): 1230-1239 © 2020 The Author(s)
Version: Author's final manuscript
ISSN
2168-2267
2168-2275
Keywords
Control and Systems Engineering, Human-Computer Interaction, Electrical and Electronic Engineering, Software, Information Systems, Computer Science Applications

Collections
  • MIT Open Access Articles

Browse

All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login

Statistics

OA StatisticsStatistics by CountryStatistics by Department
MIT Libraries
PrivacyPermissionsAccessibilityContact us
MIT
Content created by the MIT Libraries, CC BY-NC unless otherwise noted. Notify us about copyright concerns.