Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation

Lora Aroyo, Chris Welty

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Big data is having a disruptive impact across the sciences. Human annotation of semantic interpretation tasks is a critical part of big data semantics, but it is based on an antiquated ideal of a single correct truth that needs to be similarly disrupted. We expose seven myths about human annotation, most of which derive from that antiquated ideal of truth, and dispelthese myths with examples from our research. We propose a new theory of truth, crowd truth, that is based on the intuition that human interpretation is subjective, and that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.
Original languageEnglish
Pages (from-to)15-24
Number of pages10
JournalThe AI Magazine
Volume36
Issue number1
DOIs
Publication statusPublished - 1 Mar 2015

Fingerprint

Semantics
Big data

Cite this

@article{347b703d5bc243bdb5cf6100aab9cf98,
title = "Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation",
abstract = "Big data is having a disruptive impact across the sciences. Human annotation of semantic interpretation tasks is a critical part of big data semantics, but it is based on an antiquated ideal of a single correct truth that needs to be similarly disrupted. We expose seven myths about human annotation, most of which derive from that antiquated ideal of truth, and dispelthese myths with examples from our research. We propose a new theory of truth, crowd truth, that is based on the intuition that human interpretation is subjective, and that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.",
author = "Lora Aroyo and Chris Welty",
year = "2015",
month = "3",
day = "1",
doi = "10.1609/aimag.v36i1.2564",
language = "English",
volume = "36",
pages = "15--24",
journal = "The AI Magazine",
issn = "0738-4602",
publisher = "AI Access Foundation",
number = "1",

}

Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. / Aroyo, Lora; Welty, Chris.

In: The AI Magazine, Vol. 36, No. 1, 01.03.2015, p. 15-24.

Research output: Contribution to JournalArticleAcademicpeer-review

TY - JOUR

T1 - Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation

AU - Aroyo, Lora

AU - Welty, Chris

PY - 2015/3/1

Y1 - 2015/3/1

N2 - Big data is having a disruptive impact across the sciences. Human annotation of semantic interpretation tasks is a critical part of big data semantics, but it is based on an antiquated ideal of a single correct truth that needs to be similarly disrupted. We expose seven myths about human annotation, most of which derive from that antiquated ideal of truth, and dispelthese myths with examples from our research. We propose a new theory of truth, crowd truth, that is based on the intuition that human interpretation is subjective, and that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.

AB - Big data is having a disruptive impact across the sciences. Human annotation of semantic interpretation tasks is a critical part of big data semantics, but it is based on an antiquated ideal of a single correct truth that needs to be similarly disrupted. We expose seven myths about human annotation, most of which derive from that antiquated ideal of truth, and dispelthese myths with examples from our research. We propose a new theory of truth, crowd truth, that is based on the intuition that human interpretation is subjective, and that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.

UR - http://www.scopus.com/inward/record.url?scp=84927129684&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84927129684&partnerID=8YFLogxK

U2 - 10.1609/aimag.v36i1.2564

DO - 10.1609/aimag.v36i1.2564

M3 - Article

VL - 36

SP - 15

EP - 24

JO - The AI Magazine

JF - The AI Magazine

SN - 0738-4602

IS - 1

ER -