Resource Interoperability for Sustainable Benchmarking: The Case of Events

C.M. van Son; O.A. Inel; R. Morante Vallejo; L.M. Aroyo; P.T.J.M. Vossen

Resource Interoperability for Sustainable Benchmarking: The Case of Events

C.M. van Son, O.A. Inel, R. Morante Vallejo, L.M. Aroyo, P.T.J.M. Vossen

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

66 Downloads (Pure)

Abstract

With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.

Original language	English
Title of host publication	Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)
Editors	Hitoshi Isahara, Bente Maegaard, Stelios Piperidis, Christopher Cieri, Thierry Declerck, Koiti Hasida, Helene Mazo, Khalid Choukri, Sara Goggi, Joseph Mariani, Asuncion Moreno, Nicoletta Calzolari, Jan Odijk, Takenobu Tokunaga
Place of Publication	Miyazaki
Publisher	European Language Resources Association (ELRA)
Pages	1101-1111
Number of pages	11
ISBN (Electronic)	9791095546009
Publication status	Published - 2018
Event	11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan Duration: 7 May 2018 → 12 May 2018

Conference

Conference	11th International Conference on Language Resources and Evaluation, LREC 2018
Country/Territory	Japan
City	Miyazaki
Period	7/05/18 → 12/05/18

Funding

We would like to thank the anonymous reviewers for their useful feedback, everyone who kindly answered our questions and provided us with the information necessary for our analysis, and all researchers for the hard work that was put into creating the valuable resources cited in this paper. The work presented in this paper was funded by the Amsterdam Data Alliance in the QuPiD project, by the Netherlands Organization for Scientific Research (NWO) via the Spinoza grant, awarded to Piek Vossen in the project “Understanding Language by Machines”, and by CLARIAH-CORE project financed by NWO (www.clariah.nl).

Funders	Funder number
Netherlands Organization for Scientific Research
Nederlandse Organisatie voor Wetenschappelijk Onderzoek

Keywords

Annotation consistency
Events
Resource interoperability

Access to Document

https://aclanthology.org/L18-1178/

Persistent URL (handle)

Persistent link

Cite this

van Son, C. M., Inel, O. A., Morante Vallejo, R., Aroyo, L. M., & Vossen, P. T. J. M. (2018). Resource Interoperability for Sustainable Benchmarking: The Case of Events. In H. Isahara, B. Maegaard, S. Piperidis, C. Cieri, T. Declerck, K. Hasida, H. Mazo, K. Choukri, S. Goggi, J. Mariani, A. Moreno, N. Calzolari, J. Odijk, & T. Tokunaga (Eds.), Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018) (pp. 1101-1111). European Language Resources Association (ELRA). https://aclanthology.org/L18-1178/

van Son, C.M. ; Inel, O.A. ; Morante Vallejo, R. et al. / Resource Interoperability for Sustainable Benchmarking: The Case of Events. Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). editor / Hitoshi Isahara ; Bente Maegaard ; Stelios Piperidis ; Christopher Cieri ; Thierry Declerck ; Koiti Hasida ; Helene Mazo ; Khalid Choukri ; Sara Goggi ; Joseph Mariani ; Asuncion Moreno ; Nicoletta Calzolari ; Jan Odijk ; Takenobu Tokunaga. Miyazaki : European Language Resources Association (ELRA), 2018. pp. 1101-1111

@inproceedings{0dda9e0834f842d294956c95f3d5729e,

title = "Resource Interoperability for Sustainable Benchmarking: The Case of Events",

abstract = "With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.",

keywords = "Annotation consistency, Events, Resource interoperability",

author = "{van Son}, C.M. and O.A. Inel and {Morante Vallejo}, R. and L.M. Aroyo and P.T.J.M. Vossen",

year = "2018",

language = "English",

pages = "1101--1111",

editor = "Hitoshi Isahara and Bente Maegaard and Stelios Piperidis and Christopher Cieri and Thierry Declerck and Koiti Hasida and Helene Mazo and Khalid Choukri and Sara Goggi and Joseph Mariani and Asuncion Moreno and Nicoletta Calzolari and Jan Odijk and Takenobu Tokunaga",

booktitle = "Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)",

publisher = "European Language Resources Association (ELRA)",

note = "11th International Conference on Language Resources and Evaluation, LREC 2018 ; Conference date: 07-05-2018 Through 12-05-2018",

}

van Son, CM, Inel, OA, Morante Vallejo, R, Aroyo, LM & Vossen, PTJM 2018, Resource Interoperability for Sustainable Benchmarking: The Case of Events. in H Isahara, B Maegaard, S Piperidis, C Cieri, T Declerck, K Hasida, H Mazo, K Choukri, S Goggi, J Mariani, A Moreno, N Calzolari, J Odijk & T Tokunaga (eds), Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, pp. 1101-1111, 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, 7/05/18. <https://aclanthology.org/L18-1178/>

Resource Interoperability for Sustainable Benchmarking: The Case of Events. / van Son, C.M.; Inel, O.A.; Morante Vallejo, R. et al.
Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). ed. / Hitoshi Isahara; Bente Maegaard; Stelios Piperidis; Christopher Cieri; Thierry Declerck; Koiti Hasida; Helene Mazo; Khalid Choukri; Sara Goggi; Joseph Mariani; Asuncion Moreno; Nicoletta Calzolari; Jan Odijk; Takenobu Tokunaga. Miyazaki: European Language Resources Association (ELRA), 2018. p. 1101-1111.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Resource Interoperability for Sustainable Benchmarking: The Case of Events

AU - van Son, C.M.

AU - Inel, O.A.

AU - Morante Vallejo, R.

AU - Aroyo, L.M.

AU - Vossen, P.T.J.M.

PY - 2018

Y1 - 2018

N2 - With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.

AB - With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.

KW - Annotation consistency

KW - Events

KW - Resource interoperability

UR - http://www.scopus.com/inward/record.url?scp=85059886152&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85059886152&partnerID=8YFLogxK

UR - http://lrec2018.lrec-conf.org/en/

M3 - Conference contribution

AN - SCOPUS:85059886152

SP - 1101

EP - 1111

BT - Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Piperidis, Stelios

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Hasida, Koiti

A2 - Mazo, Helene

A2 - Choukri, Khalid

A2 - Goggi, Sara

A2 - Mariani, Joseph

A2 - Moreno, Asuncion

A2 - Calzolari, Nicoletta

A2 - Odijk, Jan

A2 - Tokunaga, Takenobu

PB - European Language Resources Association (ELRA)

CY - Miyazaki

T2 - 11th International Conference on Language Resources and Evaluation, LREC 2018

Y2 - 7 May 2018 through 12 May 2018

ER -

van Son CM, Inel OA, Morante Vallejo R, Aroyo LM, Vossen PTJM. Resource Interoperability for Sustainable Benchmarking: The Case of Events. In Isahara H, Maegaard B, Piperidis S, Cieri C, Declerck T, Hasida K, Mazo H, Choukri K, Goggi S, Mariani J, Moreno A, Calzolari N, Odijk J, Tokunaga T, editors, Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki: European Language Resources Association (ELRA). 2018. p. 1101-1111 Epub 2018 May 1.

Resource Interoperability for Sustainable Benchmarking: The Case of Events

Abstract

Conference

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Understanding of Language by Machines

Storylines and perspectives — Understanding of Language by Machines

Cite this

Resource Interoperability for Sustainable Benchmarking: The Case of Events

Abstract

Conference

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Projects

Understanding of Language by Machines

Storylines and perspectives — Understanding of Language by Machines

Cite this