A Generic Human–Machine Annotation Framework Based on Dynamic Cooperative Learning
Name
Zhang19-AGH_Cybernetics.pdf
Size
1.55 MB
Format
Adobe PDF
Checksum (MD5)
f506c76bda85f7b42d93749a4a166ce6
Author(s)
Zhang, Yue
Date Issued
March 2020
Journal
IEEE Transactions on Cybernetics
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Zhang, Yue, Andrea Michi, Johannes Wagner, Elisabeth Andŕe, Bj̈̈orn Schuller, Felix Weninger. "A Generic Human–Machine Annotation Framework Based on Dynamic Cooperative Learning." IEEE Transactions on Cybernetics 50 (2020): 1230-1239 © 2020 The Author(s)
Version
Author's final manuscript
Abstract
The task of obtaining meaningful annotations is atedious work, incurring considerable costs and time consumption.Dynamic active learning and cooperative learning are recentlyproposed approaches to reducing human effort of annotatingdata with subjective phenomena. In this work, we introduce anovel generic annotation framework, with the aim to achieve theoptimal trade-off between label reliability and cost reduction bymaking efficient use of human and machine work force. To thisend, we use dropout to assess model uncertainty and therebyto decide which instances can be automatically labelled by themachine and which ones require human inspection. Additionally,we propose an early stopping criterion based on inter-rateragreement in order to focus human resources on those ambiguousinstances that are difficult to label. In contrast to the existingalgorithms, the new confidence measures are not only applicableto binary classification tasks, but also regression problems. Theproposed method is evaluated on the benchmark datasets for non-native English prosody estimation, provided in the INTERSPEECHComputational Paralinguistics Challenge. In the result, the noveldynamic cooperative learning algorithm yields .424 Spearman’scorrelation coefficient compared to .413 with passive learning,while reducing the amount of human annotations by 74 %.Index Terms—Human-Machine Systems, Active Learning,Semi-supervised Learning, Confidence Measures, Inter-raterAgreement
Subjects
Control and Systems Engineering
Human-Computer Interaction
Electrical and Electronic Engineering
Software
Information Systems
Computer Science Applications
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/tcyb.2019.2901499
