Studying topical relevance with evidence-based crowdsourcing

Oana Inel, Zolt n. Szl vik, Giannis Haralabopoulos, Elena Simperl, Dan Li, Evangelos Kanoulas, Christophe Van Gysel, Lora Aroyo

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review


Information Retrieval systems rely on large test collections to measure their effectiveness in retrieving relevant documents. While the demand is high, the task of creating such test collections is laborious due to the large amounts of data that need to be annotated, and due to the intrinsic subjectivity of the task itself. In this paper we study the topical relevance from a user perspective by addressing the problems of subjectivity and ambiguity. We compare our approach and results with the established TREC annotation guidelines and results. The comparison is based on a series of crowdsourcing pilots experimenting with variables, such as relevance scale, document granularity, annotation template and the number of workers. Our results show correlation between relevance assessment accuracy and smaller document granularity, i.e., aggregation of relevance on paragraph level results in a better relevance accuracy, compared to assessment done at the level of the full document. As expected, our results also show that collecting binary relevance judgments results in a higher accuracy compared to the ternary scale used in the TREC annotation guidelines. Finally, the crowdsourced annotation tasks provided a more accurate document relevance ranking than a single assessor relevance label. This work resulted is a reliable test collection around the TREC Common Core track.

Original languageEnglish
Title of host publicationCIKM 2018 - Proceedings of the 27th ACM International Conference on Information and Knowledge Management
EditorsNorman Paton, Selcuk Candan, Haixun Wang, James Allan, Rakesh Agrawal, Alexandros Labrinidis, Alfredo Cuzzocrea, Mohammed Zaki, Divesh Srivastava, Andrei Broder, Assaf Schuster
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Electronic)9781450360142
Publication statusPublished - 17 Oct 2018
Event27th ACM International Conference on Information and Knowledge Management, CIKM 2018 - Torino, Italy
Duration: 22 Oct 201826 Oct 2018


Conference27th ACM International Conference on Information and Knowledge Management, CIKM 2018


  • Crowdsourcing
  • IR evaluation
  • TREC Common Core track


Dive into the research topics of 'Studying topical relevance with evidence-based crowdsourcing'. Together they form a unique fingerprint.

Cite this