Understanding questions expressed in natural language is a fundamental challenge studied under different applications such as question answering (QA). We explore whether recent state-of-the-art models are capable of recognizing two paraphrased questions using unsupervised learning. Firstly, we test QA models’ performance on an existing paraphrased dataset (Dev-Para). Secondly, we create a new paraphrased evaluation set (Para-SQuAD) containing multiple paraphrased question pairs from the SQuAD dataset. We describe qualitative investigations on these models and how they present paraphrased questions in continuous space. The results demonstrate that the paraphrased dataset confuses the QA models and decreases their performance. Visualizing the sentence embeddings of Para-SQuAD by the QA models suggests that all models, except BERT, struggle to recognize paraphrased questions effectively.
|Title of host publication||Artificial Intelligence and Machine Learning|
|Subtitle of host publication||32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden, The Netherlands, November 19–20, 2020, Revised Selected Papers|
|Editors||Mitra Baratchi, Lu Cao, Walter A. Kosters, Jefrey Lijffijt, Jan N. van Rijn, Frank W. Takes|
|Publisher||Springer Science and Business Media Deutschland GmbH|
|Number of pages||14|
|Publication status||Published - 2021|
|Event||32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020 - Virtual, Online|
Duration: 19 Nov 2020 → 20 Nov 2020
|Name||Communications in Computer and Information Science|
|Conference||32nd Benelux Conference on Artiﬁcial Intelligence and Belgian-Dutch Conference on Machine Learning, BNAIC/Benelearn 2020|
|Period||19/11/20 → 20/11/20|
Bibliographical noteFunding Information:
We thank the three anonymous reviewers for their constructive comments, and Michael Cochez for his feedback and helpful notes on the manuscript.
© 2021, Springer Nature Switzerland AG.
Copyright 2021 Elsevier B.V., All rights reserved.
- Natural language
- Question answering