A Generic Human–Machine Annotation Framework Based on Dynamic Cooperative Learning

Abstract

The task of obtaining meaningful annotations is atedious work, incurring considerable costs and time consumption.Dynamic active learning and cooperative learning are recentlyproposed approaches to reducing human effort of annotatingdata with subjective phenomena. In this work, we introduce anovel generic annotation framework, with the aim to achieve theoptimal trade-off between label reliability and cost reduction bymaking efficient use of human and machine work force. To thisend, we use dropout to assess model uncertainty and therebyto decide which instances can be automatically labelled by themachine and which ones require human inspection. Additionally,we propose an early stopping criterion based on inter-rateragreement in order to focus human resources on those ambiguousinstances that are difficult to label. In contrast to the existingalgorithms, the new confidence measures are not only applicableto binary classification tasks, but also regression problems. Theproposed method is evaluated on the benchmark datasets for non-native English prosody estimation, provided in the INTERSPEECHComputational Paralinguistics Challenge. In the result, the noveldynamic cooperative learning algorithm yields .424 Spearman’scorrelation coefficient compared to .413 with passive learning,while reducing the amount of human annotations by 74 %.Index Terms—Human-Machine Systems, Active Learning,Semi-supervised Learning, Confidence Measures, Inter-raterAgreement