Sequence labeling with multiple annotators
Name
Pereira_Sequence Labeling.pdf
Size
649.43 KB
Format
Adobe PDF
Checksum (MD5)
8bce157ca56affd1a1915465399517dc
Author(s)
Rodrigues, Filipe
Ribeiro, Bernardete
Pereira, Francisco C.
Date Issued
October 2013
Journal
Machine Learning
Publisher
Springer-Verlag
Citation
Rodrigues, Filipe, Francisco Pereira, and Bernardete Ribeiro. “Sequence Labeling with Multiple Annotators.” Mach Learn (October 4, 2013).
Version
Author's final manuscript
Abstract
The increasingly popular use of Crowdsourcing as a resource to obtain labeled data has been contributing to the wide awareness of the machine learning community to the problem of supervised learning from multiple annotators. Several approaches have been proposed to deal with this issue, but they disregard sequence labeling problems. However, these are very common, for example, among the Natural Language Processing and Bioinformatics communities. In this paper, we present a probabilistic approach for sequence labeling using Conditional Random Fields (CRF) for situations where label sequences from multiple annotators are available but there is no actual ground truth. The approach uses the Expectation-Maximization algorithm to jointly learn the CRF model parameters, the reliability of the annotators and the estimated ground truth. When it comes to performance, the proposed method (CRF-MA) significantly outperforms typical approaches such as majority voting.
MIT Department
Massachusetts Institute of Technology. Department of Civil and Environmental Engineering
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/s10994-013-5411-2
