Abstract
Exclusion is an important and universal linguistic skill that humans use to express what they do not want. There is little research on exclusionary retrieval, where users express what they do not want to be part of the results produced for their queries. We investigate the scenario of exclusionary retrieval in document retrieval for the first time. We present ExcluIR, a set of resources for exclusionary retrieval, consisting of an evaluation benchmark and a training set for helping retrieval models to comprehend exclusionary queries. The evaluation benchmark includes 3,452 high-quality exclusionary queries, each of which has been manually annotated. The training set contains 70,293 exclusionary queries, each paired with a positive document and a negative document. We conduct detailed experiments and analyses, obtaining three main observations: (i) existing retrieval models with different architectures struggle to comprehend exclusionary queries effectively; (ii) although integrating our training data can improve the performance of retrieval models on exclusionary retrieval, there still exists a gap compared to human performance; and (iii) generative retrieval models have a natural advantage in handling exclusionary queries.
| Original language | English |
|---|---|
| Title of host publication | AAAI'25/IAAI'25/EAAI'25: Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence |
| Editors | Toby Walsh, Julie Shah, Zico Kolter |
| Publisher | ACM |
| Pages | 13295-13303 |
| Number of pages | 9 |
| ISBN (Electronic) | 9781577358978 |
| DOIs | |
| Publication status | Published - 2025 |
| Externally published | Yes |
Funding
This work was supported by the Key R&D Program of Shandong Province with grant 2024CXGC010108, the Natural Science Foundation of China (62472261, 62102234, 62372275, 62272274, 62202271, T2293773, 62072279), the National Key R&D Program of China with grant No.2022YFC3303004, the Natural Science Foundation of Shandong Province (ZR2021QF129), and by the Dutch Research Council (NWO), under project numbers 024.004.022, NWA.1389.20.183, and KICH3.LTP.20.006, and the European Union's Horizon Europe program under grant agreement No 101070212. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
| Funders | Funder number |
|---|---|
| Nederlandse Organisatie voor Wetenschappelijk Onderzoek | KICH3.LTP.20.006, 024.004.022, NWA.1389.20.183 |
| National Key Research and Development Program of China | 2022YFC3303004 |
| European Union's Horizon Europe program | 101070212 |
| National Natural Science Foundation of China | 62202271, 62372275, T2293773, 62272274, 62472261, 62072279, 62102234 |
| Key R&D Program of Shandong Province | 2024CXGC010108 |
| Natural Science Foundation of Shandong Province | ZR2021QF129 |