TY - GEN
T1 - CrowdTruth 2.0
T2 - 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Paper 1st Workshop on Disentangling the Relation Between Crowdsourcing and Bias Management, SAD+CrowdBias 2018
AU - Dumitrache, Anca
AU - Inel, Oana
AU - Aroyo, Lora
AU - Timmermans, Benjamin
AU - Welty, Chris
PY - 2018
Y1 - 2018
N2 - Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system – worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core.
AB - Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system – worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core.
UR - http://www.scopus.com/inward/record.url?scp=85058939580&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85058939580&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85058939580
T3 - CEUR Workshop Proceedings
SP - 11
EP - 18
BT - Joint Proceedings SAD 2018 and CrowdBias 2018
A2 - Aroyo, Lora
A2 - Dumitrache, Anca
PB - CEUR-WS
Y2 - 5 July 2018
ER -