Reasoning about Ambiguous Definite Descriptions

Stefan F. Schouten; Peter Bloem; Ilia Markov; Piek Vossen

doi:10.18653/v1/2023.findings-emnlp.296

Reasoning about Ambiguous Definite Descriptions

Stefan F. Schouten, Peter Bloem, Ilia Markov, Piek Vossen

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics: EMNLP 2023
Subtitle of host publication	[Proceedings]
Publisher	Association for Computational Linguistics (ACL)
Pages	4479-4484
Number of pages	6
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.296
Publication status	Published - 2023
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Singapore
Period	6/12/23 → 10/12/23

Bibliographical note

Funding Information:
This research was supported by Huawei Finland through the DreamsLab project. All content represented the opinions of the authors, which were not necessarily shared or endorsed by their respective employers and/or sponsors.

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Funding

This research was supported by Huawei Finland through the DreamsLab project. All content represented the opinions of the authors, which were not necessarily shared or endorsed by their respective employers and/or sponsors.

Access to Document

10.18653/v1/2023.findings-emnlp.296Licence: CC BY

https://aclanthology.org/2023.findings-emnlp.296/

Persistent URL (handle)

Persistent link

Cite this

@inproceedings{3ee73fea8c714d97a7fb53304a746743,

title = "Reasoning about Ambiguous Definite Descriptions",

abstract = "Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity.",

author = "Schouten, \{Stefan F.\} and Peter Bloem and Ilia Markov and Piek Vossen",

note = "Funding Information: This research was supported by Huawei Finland through the DreamsLab project. All content represented the opinions of the authors, which were not necessarily shared or endorsed by their respective employers and/or sponsors. Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.296",

language = "English",

pages = "4479--4484",

booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

address = "United States",

}

Schouten, SF, Bloem, P , Markov, I & Vossen, P 2023, Reasoning about Ambiguous Definite Descriptions. in Findings of the Association for Computational Linguistics: EMNLP 2023: [Proceedings]. Association for Computational Linguistics (ACL), pp. 4479-4484, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.296

Reasoning about Ambiguous Definite Descriptions. / Schouten, Stefan F.; Bloem, Peter ; Markov, Ilia et al.
Findings of the Association for Computational Linguistics: EMNLP 2023: [Proceedings]. Association for Computational Linguistics (ACL), 2023. p. 4479-4484.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Reasoning about Ambiguous Definite Descriptions

AU - Schouten, Stefan F.

AU - Bloem, Peter

AU - Markov, Ilia

AU - Vossen, Piek

N1 - Funding Information: This research was supported by Huawei Finland through the DreamsLab project. All content represented the opinions of the authors, which were not necessarily shared or endorsed by their respective employers and/or sponsors. Publisher Copyright: © 2023 Association for Computational Linguistics.

PY - 2023

Y1 - 2023

N2 - Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity.

AB - Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity.

UR - https://www.scopus.com/pages/publications/85183306840

UR - https://www.scopus.com/inward/citedby.url?scp=85183306840&partnerID=8YFLogxK

U2 - 10.18653/v1/2023.findings-emnlp.296

DO - 10.18653/v1/2023.findings-emnlp.296

M3 - Conference contribution

AN - SCOPUS:85183306840

SP - 4479

EP - 4484

BT - Findings of the Association for Computational Linguistics: EMNLP 2023

PB - Association for Computational Linguistics (ACL)

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

Y2 - 6 December 2023 through 10 December 2023

ER -

Reasoning about Ambiguous Definite Descriptions

Abstract

Conference

Bibliographical note

Funding

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Reasoning about Ambiguous Definite Descriptions

Cite this