Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report)

Myrthe Reuver; Nicolas Mattis

Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report)

Myrthe Reuver
, Nicolas Mattis

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Diversity in news recommendation is important for democratic debate. Current recommendation strategies, as well as evaluation metrics for recommender systems, do not explicitly focus on this aspect of news recommendation. In the 2021 Embeddia Hackathon, we implemented one novel, normative theory-based evaluation metric, “activation”, and use it to compare two recommendation strategies of New York Times comments, one based on user likes and another on editor picks. We found that both comment recommendation strategies lead to recommendations consistently less activating than the available comments in the pool of data, but the editor’s picks more so. This might indicate that New York Times editors’ support a deliberative democratic model, in which less activation is deemed ideal for democratic debate.

Original language	English
Title of host publication	Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation
Editors	Hannu Toivonen, Michele Boggia
Publisher	Association of Computational Linguistics
Pages	134-139
Number of pages	6
ISBN (Electronic)	9781954085138
Publication status	Published - Apr 2021
Event	EACL Hackashop on News Media Content Analysis and Automated Report Generation (co-located at EACL 2021, online) - Duration: 19 Apr 2021 → 19 Apr 2021 http://embeddia.eu/hackashop2021/

Conference

Conference	EACL Hackashop on News Media Content Analysis and Automated Report Generation (co-located at EACL 2021, online)
Period	19/04/21 → 19/04/21
Internet address	http://embeddia.eu/hackashop2021/

Bibliographical note

Funding Information:
This research is funded through Open Competition Digitalization Humanities and Social Science grant nr 406.D1.19.073 awarded by the Netherlands Organization of Scientific Research (NWO). We would like to thank the hackathon organizers for organizing the event, and for excellently supporting all teams working on challenges. All remaining errors are our own.

Publisher Copyright:
© Association for Computational Linguistics

Funding

This research is funded through Open Competition Digitalization Humanities and Social Science grant nr 406.D1.19.073 awarded by the Netherlands Organization of Scientific Research (NWO). We would like to thank the hackathon organizers for organizing the event, and for excellently supporting all teams working on challenges. All remaining errors are our own.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 16 Peace, Justice and Strong Institutions

Access to Document

https://www.aclweb.org/anthology/2021.hackashop-1.19.pdfLicence: CC BY

Persistent URL (handle)

Persistent link

Cite this

Reuver, M., & Mattis, N. (2021). Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report). In H. Toivonen, & M. Boggia (Eds.), Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation (pp. 134-139). Association of Computational Linguistics. https://www.aclweb.org/anthology/2021.hackashop-1.19.pdf

@inproceedings{883ac984a2584bf19297763db1b5254d,

title = "Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report)",

abstract = "Diversity in news recommendation is important for democratic debate. Current recommendation strategies, as well as evaluation metrics for recommender systems, do not explicitly focus on this aspect of news recommendation. In the 2021 Embeddia Hackathon, we implemented one novel, normative theory-based evaluation metric, “activation”, and use it to compare two recommendation strategies of New York Times comments, one based on user likes and another on editor picks. We found that both comment recommendation strategies lead to recommendations consistently less activating than the available comments in the pool of data, but the editor{\textquoteright}s picks more so. This might indicate that New York Times editors{\textquoteright} support a deliberative democratic model, in which less activation is deemed ideal for democratic debate.",

author = "Myrthe Reuver and Nicolas Mattis",

note = "Funding Information: This research is funded through Open Competition Digitalization Humanities and Social Science grant nr 406.D1.19.073 awarded by the Netherlands Organization of Scientific Research (NWO). We would like to thank the hackathon organizers for organizing the event, and for excellently supporting all teams working on challenges. All remaining errors are our own. Publisher Copyright: {\textcopyright} Association for Computational Linguistics; EACL Hackashop on News Media Content Analysis and Automated Report Generation (co-located at EACL 2021, online) ; Conference date: 19-04-2021 Through 19-04-2021",

year = "2021",

month = apr,

language = "English",

pages = "134--139",

editor = "Hannu Toivonen and Michele Boggia",

booktitle = "Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation",

publisher = "Association of Computational Linguistics",

url = "http://embeddia.eu/hackashop2021/",

}

Reuver, M & Mattis, N 2021, Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report). in H Toivonen & M Boggia (eds), Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation. Association of Computational Linguistics, pp. 134-139, EACL Hackashop on News Media Content Analysis and Automated Report Generation (co-located at EACL 2021, online), 19/04/21. <https://www.aclweb.org/anthology/2021.hackashop-1.19.pdf>

Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report). / Reuver, Myrthe; Mattis, Nicolas.
Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation. ed. / Hannu Toivonen; Michele Boggia. Association of Computational Linguistics, 2021. p. 134-139.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report)

AU - Reuver, Myrthe

AU - Mattis, Nicolas

N1 - Funding Information: This research is funded through Open Competition Digitalization Humanities and Social Science grant nr 406.D1.19.073 awarded by the Netherlands Organization of Scientific Research (NWO). We would like to thank the hackathon organizers for organizing the event, and for excellently supporting all teams working on challenges. All remaining errors are our own. Publisher Copyright: © Association for Computational Linguistics

PY - 2021/4

Y1 - 2021/4

N2 - Diversity in news recommendation is important for democratic debate. Current recommendation strategies, as well as evaluation metrics for recommender systems, do not explicitly focus on this aspect of news recommendation. In the 2021 Embeddia Hackathon, we implemented one novel, normative theory-based evaluation metric, “activation”, and use it to compare two recommendation strategies of New York Times comments, one based on user likes and another on editor picks. We found that both comment recommendation strategies lead to recommendations consistently less activating than the available comments in the pool of data, but the editor’s picks more so. This might indicate that New York Times editors’ support a deliberative democratic model, in which less activation is deemed ideal for democratic debate.

AB - Diversity in news recommendation is important for democratic debate. Current recommendation strategies, as well as evaluation metrics for recommender systems, do not explicitly focus on this aspect of news recommendation. In the 2021 Embeddia Hackathon, we implemented one novel, normative theory-based evaluation metric, “activation”, and use it to compare two recommendation strategies of New York Times comments, one based on user likes and another on editor picks. We found that both comment recommendation strategies lead to recommendations consistently less activating than the available comments in the pool of data, but the editor’s picks more so. This might indicate that New York Times editors’ support a deliberative democratic model, in which less activation is deemed ideal for democratic debate.

UR - https://www.scopus.com/pages/publications/85115661671

UR - https://www.scopus.com/pages/publications/85115661671#tab=citedBy

UR - https://aclanthology.org/volumes/2021.hackashop-1/

M3 - Conference contribution

SP - 134

EP - 139

BT - Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation

A2 - Toivonen, Hannu

A2 - Boggia, Michele

PB - Association of Computational Linguistics

T2 - EACL Hackashop on News Media Content Analysis and Automated Report Generation (co-located at EACL 2021, online)

Y2 - 19 April 2021 through 19 April 2021

ER -

Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report)

Abstract

Conference

Bibliographical note

Funding

UN SDGs

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this