Resource Interoperability for Sustainable Benchmarking: The Case of Events

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.
LanguageEnglish
Title of host publicationProceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)
Place of PublicationMiyazaki, Japan
StatePublished - May 2018

Fingerprint

benchmarking
resource
empirical analysis
document

Cite this

van Son, C. M., Inel, O. A., Morante Vallejo, R., Aroyo, L. M., & Vossen, P. T. J. M. (2018). Resource Interoperability for Sustainable Benchmarking: The Case of Events. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018) Miyazaki, Japan.
van Son, C.M. ; Inel, O.A. ; Morante Vallejo, R. ; Aroyo, L.M. ; Vossen, P.T.J.M./ Resource Interoperability for Sustainable Benchmarking: The Case of Events. Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan, 2018.
@inproceedings{0dda9e0834f842d294956c95f3d5729e,
title = "Resource Interoperability for Sustainable Benchmarking: The Case of Events",
abstract = "With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.",
author = "{van Son}, C.M. and O.A. Inel and {Morante Vallejo}, R. and L.M. Aroyo and P.T.J.M. Vossen",
year = "2018",
month = "5",
language = "English",
booktitle = "Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)",

}

van Son, CM, Inel, OA, Morante Vallejo, R, Aroyo, LM & Vossen, PTJM 2018, Resource Interoperability for Sustainable Benchmarking: The Case of Events. in Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan.

Resource Interoperability for Sustainable Benchmarking: The Case of Events. / van Son, C.M.; Inel, O.A.; Morante Vallejo, R.; Aroyo, L.M.; Vossen, P.T.J.M.

Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan, 2018.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Resource Interoperability for Sustainable Benchmarking: The Case of Events

AU - van Son,C.M.

AU - Inel,O.A.

AU - Morante Vallejo,R.

AU - Aroyo,L.M.

AU - Vossen,P.T.J.M.

PY - 2018/5

Y1 - 2018/5

N2 - With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.

AB - With the continuous growth of benchmark corpora, which often annotate the same documents, there is a range of opportunities to compare and combine similar and complementary annotations. However, these opportunities are hampered by a wide range of problems that are related to the lack of resource interoperability. In this paper, we illustrate these problems by assessing aspects of interoperability at the document-level across a set of 20 corpora annotated with (aspects of) events. The issues range from applying different document naming conventions, to mismatches in textual content and structural/conceptual differences among annotation schemes. We provide insight into the exact document intersections between the corpora by mapping their document identifiers and perform an empirical analysis of event annotations showing their compatibility and consistency in and across the corpora. This way, we aim to make the community more aware of the challenges and opportunities and to inspire working collaboratively towards interoperable resources.

M3 - Conference contribution

BT - Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)

CY - Miyazaki, Japan

ER -

van Son CM, Inel OA, Morante Vallejo R, Aroyo LM, Vossen PTJM. Resource Interoperability for Sustainable Benchmarking: The Case of Events. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan. 2018.