Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus

Ryan Brate; Andrei Nesterov; Valentin Vogelmann; Jacco Van Ossenbruggen; Laura Hollink; Marieke Van Erp

doi:10.1145/3460210.3493553

Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus

Ryan Brate, Andrei Nesterov, Valentin Vogelmann, Jacco Van Ossenbruggen, Laura Hollink, Marieke Van Erp

Business Web and Media

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Recent initiatives by cultural heritage institutions in addressing outdated and offensive language used in their collections demonstrate the need for further understanding into when terms are problematic or contentious. This paper presents an annotated dataset of 2,715 unique samples of terms in context, drawn from a historical newspaper archive, collating 21,800 annotations of contentiousness from expert and crowd workers. We describe the contents of the corpus by analysing inter-rater agreement and differences between experts and crowd workers. In addition, we demonstrate the potential of the corpus for automated detection of contentiousness. We show that a simple classifier applied to the embedding representation of a target word provides a better than baseline performance in predicting contentiousness. We find that the term itself and the context play a role in whether a term is considered contentious.

Original language	English
Title of host publication	K-CAP '21
Subtitle of host publication	Proceedings of the 11th on Knowledge Capture Conference
Publisher	Association for Computing Machinery, Inc
Pages	17-24
Number of pages	8
ISBN (Electronic)	9781450384575
DOIs	https://doi.org/10.1145/3460210.3493553
Publication status	Published - Dec 2021
Event	11th ACM International Conference on Knowledge Capture, K-CAP 2021 - Virtual, Online, United States Duration: 2 Dec 2021 → 3 Dec 2021

Conference

Conference	11th ACM International Conference on Knowledge Capture, K-CAP 2021
Country/Territory	United States
City	Virtual, Online
Period	2/12/21 → 3/12/21

Bibliographical note

Funding Information:
This work was funded by the EuropeanaTech Challenge for Euro-peana Artificial Intelligence and Machine Learning datasets, ‘Culturally Aware AI’ funded by NWO, and SABIO funded by the Dutch Digital Heritage Network. The authors would like to thank the Cultural AI Lab and KNAW HuC colleagues for their comments and annotations and the anonymous Prolific annotators. Special thanks to Mirjam Cuper (National Library of the Netherlands) for guiding KB and Europeana procedures, Lynda Hardman (CWI) for the suggestions on the article editing, and the anonymous reviewers for their constructive feedback.

Publisher Copyright:
© 2021 ACM.

Keywords

bias
crowdsourcing
datasets
knowledge capture

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1145/3460210.3493553

Persistent URL (handle)

Copy link address

Cite this

@inproceedings{264a94e7c05e4f19861abf087fa057ee,

title = "Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus",

abstract = "Recent initiatives by cultural heritage institutions in addressing outdated and offensive language used in their collections demonstrate the need for further understanding into when terms are problematic or contentious. This paper presents an annotated dataset of 2,715 unique samples of terms in context, drawn from a historical newspaper archive, collating 21,800 annotations of contentiousness from expert and crowd workers. We describe the contents of the corpus by analysing inter-rater agreement and differences between experts and crowd workers. In addition, we demonstrate the potential of the corpus for automated detection of contentiousness. We show that a simple classifier applied to the embedding representation of a target word provides a better than baseline performance in predicting contentiousness. We find that the term itself and the context play a role in whether a term is considered contentious. ",

keywords = "bias, crowdsourcing, datasets, knowledge capture",

author = "Ryan Brate and Andrei Nesterov and Valentin Vogelmann and {Van Ossenbruggen}, Jacco and Laura Hollink and {Van Erp}, Marieke",

note = "Funding Information: This work was funded by the EuropeanaTech Challenge for Euro-peana Artificial Intelligence and Machine Learning datasets, {\textquoteleft}Culturally Aware AI{\textquoteright} funded by NWO, and SABIO funded by the Dutch Digital Heritage Network. The authors would like to thank the Cultural AI Lab and KNAW HuC colleagues for their comments and annotations and the anonymous Prolific annotators. Special thanks to Mirjam Cuper (National Library of the Netherlands) for guiding KB and Europeana procedures, Lynda Hardman (CWI) for the suggestions on the article editing, and the anonymous reviewers for their constructive feedback. Publisher Copyright: {\textcopyright} 2021 ACM.; 11th ACM International Conference on Knowledge Capture, K-CAP 2021 ; Conference date: 02-12-2021 Through 03-12-2021",

year = "2021",

month = dec,

doi = "10.1145/3460210.3493553",

language = "English",

pages = "17--24",

booktitle = "K-CAP '21",

publisher = "Association for Computing Machinery, Inc",

}

Brate, R, Nesterov, A, Vogelmann, V, Van Ossenbruggen, J, Hollink, L & Van Erp, M 2021, Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus. in K-CAP '21: Proceedings of the 11th on Knowledge Capture Conference. Association for Computing Machinery, Inc, pp. 17-24, 11th ACM International Conference on Knowledge Capture, K-CAP 2021, Virtual, Online, United States, 2/12/21. https://doi.org/10.1145/3460210.3493553

Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus. / Brate, Ryan; Nesterov, Andrei; Vogelmann, Valentin et al.
K-CAP '21: Proceedings of the 11th on Knowledge Capture Conference. Association for Computing Machinery, Inc, 2021. p. 17-24.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus

AU - Brate, Ryan

AU - Nesterov, Andrei

AU - Vogelmann, Valentin

AU - Van Ossenbruggen, Jacco

AU - Hollink, Laura

AU - Van Erp, Marieke

N1 - Funding Information: This work was funded by the EuropeanaTech Challenge for Euro-peana Artificial Intelligence and Machine Learning datasets, ‘Culturally Aware AI’ funded by NWO, and SABIO funded by the Dutch Digital Heritage Network. The authors would like to thank the Cultural AI Lab and KNAW HuC colleagues for their comments and annotations and the anonymous Prolific annotators. Special thanks to Mirjam Cuper (National Library of the Netherlands) for guiding KB and Europeana procedures, Lynda Hardman (CWI) for the suggestions on the article editing, and the anonymous reviewers for their constructive feedback. Publisher Copyright: © 2021 ACM.

PY - 2021/12

Y1 - 2021/12

N2 - Recent initiatives by cultural heritage institutions in addressing outdated and offensive language used in their collections demonstrate the need for further understanding into when terms are problematic or contentious. This paper presents an annotated dataset of 2,715 unique samples of terms in context, drawn from a historical newspaper archive, collating 21,800 annotations of contentiousness from expert and crowd workers. We describe the contents of the corpus by analysing inter-rater agreement and differences between experts and crowd workers. In addition, we demonstrate the potential of the corpus for automated detection of contentiousness. We show that a simple classifier applied to the embedding representation of a target word provides a better than baseline performance in predicting contentiousness. We find that the term itself and the context play a role in whether a term is considered contentious.

AB - Recent initiatives by cultural heritage institutions in addressing outdated and offensive language used in their collections demonstrate the need for further understanding into when terms are problematic or contentious. This paper presents an annotated dataset of 2,715 unique samples of terms in context, drawn from a historical newspaper archive, collating 21,800 annotations of contentiousness from expert and crowd workers. We describe the contents of the corpus by analysing inter-rater agreement and differences between experts and crowd workers. In addition, we demonstrate the potential of the corpus for automated detection of contentiousness. We show that a simple classifier applied to the embedding representation of a target word provides a better than baseline performance in predicting contentiousness. We find that the term itself and the context play a role in whether a term is considered contentious.

KW - bias

KW - crowdsourcing

KW - datasets

KW - knowledge capture

UR - http://www.scopus.com/inward/record.url?scp=85120848706&partnerID=8YFLogxK

U2 - 10.1145/3460210.3493553

DO - 10.1145/3460210.3493553

M3 - Conference contribution

AN - SCOPUS:85120848706

SP - 17

EP - 24

BT - K-CAP '21

PB - Association for Computing Machinery, Inc

T2 - 11th ACM International Conference on Knowledge Capture, K-CAP 2021

Y2 - 2 December 2021 through 3 December 2021

ER -

Capturing Contentiousness: Constructing the Contentious Terms in Context Corpus

Abstract

Conference

Bibliographical note

Keywords

UN SDGs

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this