Quantifying retrieval bias in Web archive search

Thaer Samar, Myriam C. Traub, Jacco van Ossenbruggen, Lynda Hardman, Arjen P. de Vries*

*Corresponding author for this work

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

A Web archive usually contains multiple versions of documents crawled from the Web at different points in time. One possible way for users to access a Web archive is through full-text search systems. However, previous studies have shown that these systems can induce a bias, known as the retrievability bias, on the accessibility of documents in community-collected collections (such as TREC collections). This bias can be measured by analyzing the distribution of the retrievability scores for each document in a collection, quantifying the likelihood of a document’s retrieval. We investigate the suitability of retrievability scores in retrieval systems that consider every version of a document in a Web archive as an independent document. We show that the retrievability of documents can vary for different versions of the same document and that retrieval systems induce biases to different extents. We quantify this bias for a retrieval system which is adapted to handle multiple versions of the same document. The retrieval system indexes each version of a document independently, and we refine the search results using two techniques to aggregate similar versions. The first approach is to collapse similar versions of a document based on content similarity. The second approach is to collapse all versions of the same document based on their URLs. In both cases, we found that the degree of bias is related to the aggregation level of versions of the same document. Finally, we study the effect of bias across time using the retrievability measure. Specifically, we investigate whether the number of documents crawled in a particular year correlates with the number of documents in the search results from that year. Assuming queries are not inherently temporal in nature, the analysis is based on the timestamps of documents in the search results returned using the retrieval model for all queries. The results show a relation between the number of documents per year and the number of documents retrieved by the retrieval system from that year. We further investigated the relation between the queries’ timestamps and the documents’ timestamps. First, we split the queries into different time frames using a 1-year granularity. Then, we issued the queries against the retrieval system. The results show that temporal queries indeed retrieve more documents from the assumed time frame. Thus, the documents from the same time frame were preferred by the retrieval system over documents from other time frames.

Original languageEnglish
Pages (from-to)57-75
Number of pages19
JournalInternational Journal on Digital Libraries
Volume19
Issue number1
Early online date14 Nov 2017
DOIs
Publication statusPublished - 1 Mar 2018

Funding

Acknowledgements This research was supported by the Netherlands Organization for Scientific Research (WebART project, NWO CATCH #640.005.001), and the Dutch COMMIT/program (SEALINCMedia project). We would like to thank the National Library of the Netherlands for their support. Part of the analysis work was carried out on the Dutch national e-infrastructure with the support of the SURF Foundation. We are grateful for the input from our colleagues Jiyin He and Desmond Elliott. This research was supported by the Netherlands Organization for Scientific Research (WebART project, NWO CATCH #640.005.001), and the Dutch COMMIT/program (SEALINCMedia project). We would like to thank the National Library of the Netherlands for their support. Part of the analysis work was carried out on the Dutch national e-infrastructure with the support of the SURF Foundation. We are grateful for the input from our colleagues Jiyin He and Desmond Elliott.

FundersFunder number
Dutch COMMIT/program
NWO CATCHCATCH #640.005.001
Netherlands Organization for Scientific Research
SEALINCMedia
SURF Foundation
Nederlandse Organisatie voor Wetenschappelijk Onderzoek640.005.001

    Keywords

    • Evaluation
    • Retrieval bias
    • Web archive

    Fingerprint

    Dive into the research topics of 'Quantifying retrieval bias in Web archive search'. Together they form a unique fingerprint.

    Cite this