Resource Interoperability for Sustainable Benchmarking: The Case of Events

C.M. van Son, O.A. Inel, R. Morante Vallejo, L.M. Aroyo, P.T.J.M. Vossen

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

66 Downloads (Pure)


With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.
Original languageEnglish
Title of host publicationProceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)
EditorsHitoshi Isahara, Bente Maegaard, Stelios Piperidis, Christopher Cieri, Thierry Declerck, Koiti Hasida, Helene Mazo, Khalid Choukri, Sara Goggi, Joseph Mariani, Asuncion Moreno, Nicoletta Calzolari, Jan Odijk, Takenobu Tokunaga
Place of PublicationMiyazaki
PublisherEuropean Language Resources Association (ELRA)
Number of pages11
ISBN (Electronic)9791095546009
Publication statusPublished - 2018
Event11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan
Duration: 7 May 201812 May 2018


Conference11th International Conference on Language Resources and Evaluation, LREC 2018


We would like to thank the anonymous reviewers for their useful feedback, everyone who kindly answered our questions and provided us with the information necessary for our analysis, and all researchers for the hard work that was put into creating the valuable resources cited in this paper. The work presented in this paper was funded by the Amsterdam Data Alliance in the QuPiD project, by the Netherlands Organization for Scientific Research (NWO) via the Spinoza grant, awarded to Piek Vossen in the project “Understanding Language by Machines”, and by CLARIAH-CORE project financed by NWO (

FundersFunder number
Netherlands Organization for Scientific Research
Nederlandse Organisatie voor Wetenschappelijk Onderzoek


    • Annotation consistency
    • Events
    • Resource interoperability


    Dive into the research topics of 'Resource Interoperability for Sustainable Benchmarking: The Case of Events'. Together they form a unique fingerprint.

    Cite this