Abstract
We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing |
| Editors | Houda Bouamor, Juan Pino, Kalika Bali |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 10728-10752 |
| Number of pages | 25 |
| ISBN (Electronic) | 9798891760608 |
| DOIs | |
| Publication status | Published - 2023 |
| Event | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023 |
Conference
| Conference | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 |
|---|---|
| Country/Territory | Singapore |
| City | Hybrid, Singapore |
| Period | 6/12/23 → 10/12/23 |
Bibliographical note
Publisher Copyright:©2023 Association for Computational Linguistics.
Funding
Many thanks go to our supporting annotators: Maria Francis, Christoph Otto, and Anna Spasiano. We would also like to thank Alexander Koller, Matthias Lindemann, Ivan Titov, and Juri Opitz for insightful discussions. Thanks also to Aditya Surikuchi and Sandro Pezelle for feedback on the paper. Last but not least, we would like to thank the reviewers for their helpful comments. This work is funded in part by the Deutsche Forschungsge-meinschaft (DFG, German Research Foundation) – 492792184. This work is also part of the research programme Learning meaning from structure: neural semantic parsing with minimalist grammars with project number VI.Veni.194.057, which is funded by the Dutch Research Council (NWO). Many thanks go to our supporting annotators: Maria Francis, Christoph Otto, and Anna Spasiano. We would also like to thank Alexander Koller, Matthias Lindemann, Ivan Titov, and Juri Opitz for insightful discussions. Thanks also to Aditya Surikuchi and Sandro Pezelle for feedback on the paper. Last but not least, we would like to thank the reviewers for their helpful comments. This work is funded in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 492792184. This work is also part of the research programme Learning meaning from structure: neural semantic parsing with minimalist grammars with project number VI.Veni.194.057, which is funded by the Dutch Research Council (NWO).
| Funders | Funder number |
|---|---|
| Alexander Koller | |
| Deutsche Forschungsgemeinschaft | VI.Veni.194.057, 492792184 |
| Nederlandse Organisatie voor Wetenschappelijk Onderzoek |