Crowdsourcing ground truth for Question Answering using CrowdTruth

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Gathering training and evaluation data for open domain tasks, such as general question answering, is a challenging task. Typically, ground truth data is provided by human expert annotators, however, in an open domain experts are difficult to define. Moreover, the overall process for annotating examples can be lengthy and expensive. Naturally, crowdsourcing has become a mainstream approach for filling this gap, i.e. gathering human interpretation data. However, similar to the traditional expert annotation tasks, most of those methods use majority voting to measure the quality of the annotations and thus aim at identifying a single right answer for each example, despite the fact that many annotation tasks can have multiple interpretations, which results in multiple correct answers to the same question. We present a crowdsourcing-based approach for efficiently gathering ground truth data called CrowdTruth, where disagreement-based metrics are used to harness the multitude of human interpretation and measure the quality of the resulting ground truth. We exemplify our approach in two semantic interpretation use cases for answering questions.
Original languageEnglish
Title of host publicationProceedings of the ACM WebScience conference
Place of PublicationOxford
PublisherACM
ISBN (Print)9781450336727
DOIs
Publication statusPublished - 2015
EventWebSci ’15 - Oxford
Duration: 28 Jun 20151 Jul 2015

Conference

ConferenceWebSci ’15
Period28/06/151/07/15

Fingerprint

Semantics

Cite this

Timmermans, B.F.L. ; Aroyo, L.M. ; Welty, C.A. / Crowdsourcing ground truth for Question Answering using CrowdTruth. Proceedings of the ACM WebScience conference. Oxford : ACM, 2015.
@inproceedings{710dc49e74f7491ba1d77f060455489a,
title = "Crowdsourcing ground truth for Question Answering using CrowdTruth",
abstract = "Gathering training and evaluation data for open domain tasks, such as general question answering, is a challenging task. Typically, ground truth data is provided by human expert annotators, however, in an open domain experts are difficult to define. Moreover, the overall process for annotating examples can be lengthy and expensive. Naturally, crowdsourcing has become a mainstream approach for filling this gap, i.e. gathering human interpretation data. However, similar to the traditional expert annotation tasks, most of those methods use majority voting to measure the quality of the annotations and thus aim at identifying a single right answer for each example, despite the fact that many annotation tasks can have multiple interpretations, which results in multiple correct answers to the same question. We present a crowdsourcing-based approach for efficiently gathering ground truth data called CrowdTruth, where disagreement-based metrics are used to harness the multitude of human interpretation and measure the quality of the resulting ground truth. We exemplify our approach in two semantic interpretation use cases for answering questions.",
author = "B.F.L. Timmermans and L.M. Aroyo and C.A. Welty",
year = "2015",
doi = "10.1145/2786451.2786492",
language = "English",
isbn = "9781450336727",
booktitle = "Proceedings of the ACM WebScience conference",
publisher = "ACM",

}

Timmermans, BFL, Aroyo, LM & Welty, CA 2015, Crowdsourcing ground truth for Question Answering using CrowdTruth. in Proceedings of the ACM WebScience conference. ACM, Oxford, WebSci ’15, 28/06/15. https://doi.org/10.1145/2786451.2786492

Crowdsourcing ground truth for Question Answering using CrowdTruth. / Timmermans, B.F.L.; Aroyo, L.M.; Welty, C.A.

Proceedings of the ACM WebScience conference. Oxford : ACM, 2015.

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Crowdsourcing ground truth for Question Answering using CrowdTruth

AU - Timmermans, B.F.L.

AU - Aroyo, L.M.

AU - Welty, C.A.

PY - 2015

Y1 - 2015

N2 - Gathering training and evaluation data for open domain tasks, such as general question answering, is a challenging task. Typically, ground truth data is provided by human expert annotators, however, in an open domain experts are difficult to define. Moreover, the overall process for annotating examples can be lengthy and expensive. Naturally, crowdsourcing has become a mainstream approach for filling this gap, i.e. gathering human interpretation data. However, similar to the traditional expert annotation tasks, most of those methods use majority voting to measure the quality of the annotations and thus aim at identifying a single right answer for each example, despite the fact that many annotation tasks can have multiple interpretations, which results in multiple correct answers to the same question. We present a crowdsourcing-based approach for efficiently gathering ground truth data called CrowdTruth, where disagreement-based metrics are used to harness the multitude of human interpretation and measure the quality of the resulting ground truth. We exemplify our approach in two semantic interpretation use cases for answering questions.

AB - Gathering training and evaluation data for open domain tasks, such as general question answering, is a challenging task. Typically, ground truth data is provided by human expert annotators, however, in an open domain experts are difficult to define. Moreover, the overall process for annotating examples can be lengthy and expensive. Naturally, crowdsourcing has become a mainstream approach for filling this gap, i.e. gathering human interpretation data. However, similar to the traditional expert annotation tasks, most of those methods use majority voting to measure the quality of the annotations and thus aim at identifying a single right answer for each example, despite the fact that many annotation tasks can have multiple interpretations, which results in multiple correct answers to the same question. We present a crowdsourcing-based approach for efficiently gathering ground truth data called CrowdTruth, where disagreement-based metrics are used to harness the multitude of human interpretation and measure the quality of the resulting ground truth. We exemplify our approach in two semantic interpretation use cases for answering questions.

U2 - 10.1145/2786451.2786492

DO - 10.1145/2786451.2786492

M3 - Conference contribution

SN - 9781450336727

BT - Proceedings of the ACM WebScience conference

PB - ACM

CY - Oxford

ER -