Evaluating the Robustness of Question-Answering Models to Paraphrased Questions

Paulo Alting von Geusau; Peter Bloem

doi:10.1007/978-3-030-76640-5_1

Evaluating the Robustness of Question-Answering Models to Paraphrased Questions

Paulo Alting von Geusau^*, Peter Bloem

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

75 Downloads (Pure)

Abstract

Understanding questions expressed in natural language is a fundamental challenge studied under different applications such as question answering (QA). We explore whether recent state-of-the-art models are capable of recognizing two paraphrased questions using unsupervised learning. Firstly, we test QA models’ performance on an existing paraphrased dataset (Dev-Para). Secondly, we create a new paraphrased evaluation set (Para-SQuAD) containing multiple paraphrased question pairs from the SQuAD dataset. We describe qualitative investigations on these models and how they present paraphrased questions in continuous space. The results demonstrate that the paraphrased dataset confuses the QA models and decreases their performance. Visualizing the sentence embeddings of Para-SQuAD by the QA models suggests that all models, except BERT, struggle to recognize paraphrased questions effectively.

Original language	English
Title of host publication	Artificial Intelligence and Machine Learning
Subtitle of host publication	32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers
Editors	Mitra Baratchi, Lu Cao, Walter A. Kosters, Jefrey Lijffijt, Jan N. van Rijn, Frank W. Takes
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	1-14
Number of pages	14
ISBN (Electronic)	9783030766405
ISBN (Print)	9783030766399
DOIs	https://doi.org/10.1007/978-3-030-76640-5_1
Publication status	Published - 2021
Event	32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020 - Virtual, Online Duration: 19 Nov 2020 → 20 Nov 2020

Publication series

Name	Communications in Computer and Information Science
Volume	1398 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020
City	Virtual, Online
Period	19/11/20 → 20/11/20

Bibliographical note

Funding Information:
We thank the three anonymous reviewers for their constructive comments, and Michael Cochez for his feedback and helpful notes on the manuscript.

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.

Keywords

Embeddings
Natural language
Question answering
Transformers

Access to Document

10.1007/978-3-030-76640-5_1

Evaluating the Robustness of Question-Answering Models to Paraphrased QuestionsFinal published version, 1.02 MBLicence: Article 25fa Dutch Copyright Act

Persistent URL (handle)

Copy link address

Cite this

Alting von Geusau, P., & Bloem, P. (2021). Evaluating the Robustness of Question-Answering Models to Paraphrased Questions. In M. Baratchi, L. Cao, W. A. Kosters, J. Lijffijt, J. N. van Rijn, & F. W. Takes (Eds.), Artificial Intelligence and Machine Learning: 32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers (pp. 1-14). (Communications in Computer and Information Science; Vol. 1398 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-76640-5_1

Alting von Geusau, Paulo ; Bloem, Peter. / Evaluating the Robustness of Question-Answering Models to Paraphrased Questions. Artificial Intelligence and Machine Learning: 32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers. editor / Mitra Baratchi ; Lu Cao ; Walter A. Kosters ; Jefrey Lijffijt ; Jan N. van Rijn ; Frank W. Takes. Springer Science and Business Media Deutschland GmbH, 2021. pp. 1-14 (Communications in Computer and Information Science).

@inproceedings{3f24f257ed8a48c587b5f7a20883696a,

title = "Evaluating the Robustness of Question-Answering Models to Paraphrased Questions",

abstract = "Understanding questions expressed in natural language is a fundamental challenge studied under different applications such as question answering (QA). We explore whether recent state-of-the-art models are capable of recognizing two paraphrased questions using unsupervised learning. Firstly, we test QA models{\textquoteright} performance on an existing paraphrased dataset (Dev-Para). Secondly, we create a new paraphrased evaluation set (Para-SQuAD) containing multiple paraphrased question pairs from the SQuAD dataset. We describe qualitative investigations on these models and how they present paraphrased questions in continuous space. The results demonstrate that the paraphrased dataset confuses the QA models and decreases their performance. Visualizing the sentence embeddings of Para-SQuAD by the QA models suggests that all models, except BERT, struggle to recognize paraphrased questions effectively.",

keywords = "Embeddings, Natural language, Question answering, Transformers",

author = "{Alting von Geusau}, Paulo and Peter Bloem",

note = "Funding Information: We thank the three anonymous reviewers for their constructive comments, and Michael Cochez for his feedback and helpful notes on the manuscript. Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; 32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020 ; Conference date: 19-11-2020 Through 20-11-2020",

year = "2021",

doi = "10.1007/978-3-030-76640-5_1",

language = "English",

isbn = "9783030766399",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "1--14",

editor = "Mitra Baratchi and Lu Cao and Kosters, {Walter A.} and Jefrey Lijffijt and {van Rijn}, {Jan N.} and Takes, {Frank W.}",

booktitle = "Artificial Intelligence and Machine Learning",

address = "Germany",

}

Alting von Geusau, P & Bloem, P 2021, Evaluating the Robustness of Question-Answering Models to Paraphrased Questions. in M Baratchi, L Cao, WA Kosters, J Lijffijt, JN van Rijn & FW Takes (eds), Artificial Intelligence and Machine Learning: 32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers. Communications in Computer and Information Science, vol. 1398 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 1-14, 32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020, Virtual, Online, 19/11/20. https://doi.org/10.1007/978-3-030-76640-5_1

Evaluating the Robustness of Question-Answering Models to Paraphrased Questions. / Alting von Geusau, Paulo; Bloem, Peter.
Artificial Intelligence and Machine Learning: 32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers. ed. / Mitra Baratchi; Lu Cao; Walter A. Kosters; Jefrey Lijffijt; Jan N. van Rijn; Frank W. Takes. Springer Science and Business Media Deutschland GmbH, 2021. p. 1-14 (Communications in Computer and Information Science; Vol. 1398 CCIS).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Evaluating the Robustness of Question-Answering Models to Paraphrased Questions

AU - Alting von Geusau, Paulo

AU - Bloem, Peter

N1 - Funding Information: We thank the three anonymous reviewers for their constructive comments, and Michael Cochez for his feedback and helpful notes on the manuscript. Publisher Copyright: © 2021, Springer Nature Switzerland AG. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021

Y1 - 2021

N2 - Understanding questions expressed in natural language is a fundamental challenge studied under different applications such as question answering (QA). We explore whether recent state-of-the-art models are capable of recognizing two paraphrased questions using unsupervised learning. Firstly, we test QA models’ performance on an existing paraphrased dataset (Dev-Para). Secondly, we create a new paraphrased evaluation set (Para-SQuAD) containing multiple paraphrased question pairs from the SQuAD dataset. We describe qualitative investigations on these models and how they present paraphrased questions in continuous space. The results demonstrate that the paraphrased dataset confuses the QA models and decreases their performance. Visualizing the sentence embeddings of Para-SQuAD by the QA models suggests that all models, except BERT, struggle to recognize paraphrased questions effectively.

AB - Understanding questions expressed in natural language is a fundamental challenge studied under different applications such as question answering (QA). We explore whether recent state-of-the-art models are capable of recognizing two paraphrased questions using unsupervised learning. Firstly, we test QA models’ performance on an existing paraphrased dataset (Dev-Para). Secondly, we create a new paraphrased evaluation set (Para-SQuAD) containing multiple paraphrased question pairs from the SQuAD dataset. We describe qualitative investigations on these models and how they present paraphrased questions in continuous space. The results demonstrate that the paraphrased dataset confuses the QA models and decreases their performance. Visualizing the sentence embeddings of Para-SQuAD by the QA models suggests that all models, except BERT, struggle to recognize paraphrased questions effectively.

KW - Embeddings

KW - Natural language

KW - Question answering

KW - Transformers

UR - http://www.scopus.com/inward/record.url?scp=85111268626&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85111268626&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-76640-5_1

DO - 10.1007/978-3-030-76640-5_1

M3 - Conference contribution

AN - SCOPUS:85111268626

SN - 9783030766399

T3 - Communications in Computer and Information Science

SP - 1

EP - 14

BT - Artificial Intelligence and Machine Learning

A2 - Baratchi, Mitra

A2 - Cao, Lu

A2 - Kosters, Walter A.

A2 - Lijffijt, Jefrey

A2 - van Rijn, Jan N.

A2 - Takes, Frank W.

PB - Springer Science and Business Media Deutschland GmbH

T2 - 32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020

Y2 - 19 November 2020 through 20 November 2020

ER -

Alting von Geusau P, Bloem P. Evaluating the Robustness of Question-Answering Models to Paraphrased Questions. In Baratchi M, Cao L, Kosters WA, Lijffijt J, van Rijn JN, Takes FW, editors, Artificial Intelligence and Machine Learning: 32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2021. p. 1-14. (Communications in Computer and Information Science). Epub 2021 May 20. doi: 10.1007/978-3-030-76640-5_1

Evaluating the Robustness of Question-Answering Models to Paraphrased Questions

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this