Publication: A novel label aggregation with attenuated scores for ground-Truth identification of dataset annotation with crowdsourcing
Issued Date
2017-04-01
Resource Type
ISSN
17451361
09168532
09168532
Other identifier(s)
2-s2.0-85017700020
Rights
Mahidol University
Rights Holder(s)
SCOPUS
Bibliographic Citation
IEICE Transactions on Information and Systems. Vol.E100D, No.4 (2017), 750-757
Suggested Citation
Ratchainant Thammasudjarit, Anon Plangprasopchok, Charnyote Pluempitiwiriyawej A novel label aggregation with attenuated scores for ground-Truth identification of dataset annotation with crowdsourcing. IEICE Transactions on Information and Systems. Vol.E100D, No.4 (2017), 750-757. doi:10.1587/transinf.2016DAP0024 Retrieved from: https://repository.li.mahidol.ac.th/handle/20.500.14594/42357
Research Projects
Organizational Units
Authors
Journal Issue
Thesis
Title
A novel label aggregation with attenuated scores for ground-Truth identification of dataset annotation with crowdsourcing
Other Contributor(s)
Abstract
© 2017 The Institute of Electronics, Information and Communication Engineers. Ground-Truth identification -The process, which infers the most probable labels, for a certain dataset, from crowdsourcing annotations - is a crucial task to make the dataset usable, e.g., for a supervised learning problem. Nevertheless, the process is challenging because annotations from multiple annotators are inconsistent and noisy. Existing methods require a set of data sample with corresponding ground-Truth labels to precisely estimate annotator performance but such samples are difficult to obtain in practice. Moreover, the process requires a post-editing step to validate indefinite labels, which are generally unidentifiable without thoroughly inspecting the whole annotated data. To address the challenges, this paper introduces: 1) Attenuated score (A-score) -An indicator that locally measures annotator performance for segments of annotation sequences, and 2) label aggregation method that applies A-score for ground-Truth identification. The experimental results demonstrate that A-score label aggregation outperforms majority vote in all datasets by accurately recovering more labels. It also achieves higher F1 scores than those of the strong baselines in all multi-class data. Additionally, the results suggest that A-score is a promising indicator that helps identifying indefinite labels for the postediting procedure.