Abstract
Legal judgment assistants are developing fast due to impressive progress of large language models (LLMs). However, people can hardly trust the results generated by a model without reliable analysis of legal judgement. For legal practitioners, it is common practice to utilize syllogistic reasoning to select and evaluate the arguments of the parties as part of the legal decision-making process. But the development of syllogistic reasoning for legal judgment analysis is hindered by the lack of resources: (1) there is no large-scale syllogistic reasoning dataset for legal judgment analysis, and (2) there is no set of established benchmarks for legal judgment analysis. In this paper, we construct and manually correct a syllogistic reasoning dataset for legal judgment analysis. The dataset contains 11,239 criminal cases which cover 4 criminal elements, 80 charges and 124 articles. We also select a set of large language models as benchmarks, and conduct a in-depth analysis of the capacity of their legal judgment analysis.
| Original language | English |
|---|---|
| Title of host publication | EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings |
| Editors | H. Bouamor, J. Pino, K. Bali |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 13997-14009 |
| ISBN (Electronic) | 9798891760608 |
| Publication status | Published - 2023 |
| Externally published | Yes |
| Event | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023 |
Conference
| Conference | 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 |
|---|---|
| Country/Territory | Singapore |
| City | Hybrid, Singapore |
| Period | 6/12/23 → 10/12/23 |
Funding
We would like to thank the editors and reviewers for their helpful comments. This research was supported by the National Key R&D Program of China (No.2022YFC3303004, No.2020YFB1406704), the Natural Science Foundation of China (62102234, 62272274, 62202271, 61902219, 61972234, 62072279), the Key Scientific and Technological Innovation Program of Shandong Province (2019JZZY010129), the Fundamental Research Funds of Shandong University, and VOXReality (European Union Grant, No. 101070521). All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
| Funders | Funder number |
|---|---|
| Fundamental Research Fund of Shandong University | |
| European Commission | 101070521 |
| European Commission | |
| National Natural Science Foundation of China | 62202271, 61972234, 62272274, 61902219, 62072279, 62102234 |
| National Natural Science Foundation of China | |
| National Key Research and Development Program of China | 2020YFB1406704, 2022YFC3303004 |
| National Key Research and Development Program of China | |
| Major Scientific and Technological Innovation Project of Shandong Province | 2019JZZY010129 |
| Major Scientific and Technological Innovation Project of Shandong Province |