On the use of the descriptive variable for enhancing the aggregation of crowdsourced labels
The use of crowdsourcing for annotating data has become a popular and cheap alternative to expert labelling. As a consequence, an aggregation task is required to combine the different labels provided and agree on a single one per example. Most aggregation techniques, including the simple and robust...
| Authors: | , , |
|---|---|
| Format: | article |
| Status: | Published version |
| Publication Date: | 2022 |
| Country: | España |
| Institution: | Universidad de Barcelona |
| Repository: | Dipòsit Digital de la UB |
| OAI Identifier: | oai:diposit.ub.edu:2445/189541 |
| Online Access: | https://hdl.handle.net/2445/189541 |
| Access Level: | Open access |
| Keyword: | Aprenentatge automàtic Cultura participativa Dades massives Machine learning Participatory culture Big data |
| Summary: | The use of crowdsourcing for annotating data has become a popular and cheap alternative to expert labelling. As a consequence, an aggregation task is required to combine the different labels provided and agree on a single one per example. Most aggregation techniques, including the simple and robust majority voting¿to select the label with the largest number of votes¿disregard the descriptive information provided by the explanatory variable. In this paper, we propose domain-aware voting, an extension of majority voting which incorporates the descriptive variable and the rest of the instances of the dataset for aggregating the label of every instance. The experimental results with simulated and real-world crowdsourced data suggest that domain-aware voting is a competitive alternative to majority voting, especially when a part of the dataset is unlabelled. We elaborate on practical criteria for the use of domain-aware voting. |
|---|