Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task

Tommaso Caselli; Roser Morante

Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task

Language

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

In this article we review Temporal Processing systems that participated in the TempEval-3 task as a basis to develop our own system, that we also present and release. The system incorporates high level lexical semantic features, obtaining the best scores for event detection (F1-Class 72.24) and second best result for temporal relation classification from raw text (F1 29.69) when evaluated on the TempEval-3 data. Additionally, we analyse the errors of all TempEval-3 systems for which the output is publicly available with the purpose of finding out what are the weaknesses of current approaches. Although incorporating lexical semantics features increases the performance of our system, the error analysis shows that systems should incorporate inference mechanisms and world knowledge, as well as having strategies to compensate for data skewness.

Original language	English
Title of host publication	LREC 2018 - 11th International Conference on Language Resources and Evaluation
Editors	Hitoshi Isahara, Bente Maegaard, Stelios Piperidis, Christopher Cieri, Thierry Declerck, Koiti Hasida, Helene Mazo, Khalid Choukri, Sara Goggi, Joseph Mariani, Asuncion Moreno, Nicoletta Calzolari, Jan Odijk, Takenobu Tokunaga
Publisher	European Language Resources Association (ELRA)
Pages	339-348
Number of pages	10
ISBN (Electronic)	9791095546009
Publication status	Published - 2019
Event	11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan Duration: 7 May 2018 → 12 May 2018

Conference

Conference	11th International Conference on Language Resources and Evaluation, LREC 2018
Country	Japan
City	Miyazaki
Period	7/05/18 → 12/05/18

Keywords

Error analysis
Temporal processing
Written corpora

Access to Document

Cite this

Caselli, T., & Morante, R. (2019). Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task. In H. Isahara, B. Maegaard, S. Piperidis, C. Cieri, T. Declerck, K. Hasida, H. Mazo, K. Choukri, S. Goggi, J. Mariani, A. Moreno, N. Calzolari, J. Odijk, & T. Tokunaga (Eds.), LREC 2018 - 11th International Conference on Language Resources and Evaluation (pp. 339-348). [880] European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2018/summaries/880.html

Caselli, Tommaso ; Morante, Roser. / Systems' agreements and disagreements in temporal processing : An extensive error analysis of the TempEval-3 task. LREC 2018 - 11th International Conference on Language Resources and Evaluation. editor / Hitoshi Isahara ; Bente Maegaard ; Stelios Piperidis ; Christopher Cieri ; Thierry Declerck ; Koiti Hasida ; Helene Mazo ; Khalid Choukri ; Sara Goggi ; Joseph Mariani ; Asuncion Moreno ; Nicoletta Calzolari ; Jan Odijk ; Takenobu Tokunaga. European Language Resources Association (ELRA), 2019. pp. 339-348

@inproceedings{b569b5b42a5846fe8648aed3377d22e8,

title = "Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task",

abstract = "In this article we review Temporal Processing systems that participated in the TempEval-3 task as a basis to develop our own system, that we also present and release. The system incorporates high level lexical semantic features, obtaining the best scores for event detection (F1-Class 72.24) and second best result for temporal relation classification from raw text (F1 29.69) when evaluated on the TempEval-3 data. Additionally, we analyse the errors of all TempEval-3 systems for which the output is publicly available with the purpose of finding out what are the weaknesses of current approaches. Although incorporating lexical semantics features increases the performance of our system, the error analysis shows that systems should incorporate inference mechanisms and world knowledge, as well as having strategies to compensate for data skewness.",

keywords = "Error analysis, Temporal processing, Written corpora",

author = "Tommaso Caselli and Roser Morante",

year = "2019",

language = "English",

pages = "339--348",

editor = "Hitoshi Isahara and Bente Maegaard and Stelios Piperidis and Christopher Cieri and Thierry Declerck and Koiti Hasida and Helene Mazo and Khalid Choukri and Sara Goggi and Joseph Mariani and Asuncion Moreno and Nicoletta Calzolari and Jan Odijk and Takenobu Tokunaga",

booktitle = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",

publisher = "European Language Resources Association (ELRA)",

note = "11th International Conference on Language Resources and Evaluation, LREC 2018 ; Conference date: 07-05-2018 Through 12-05-2018",

}

Caselli, T & Morante, R 2019, Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task. in H Isahara, B Maegaard, S Piperidis, C Cieri, T Declerck, K Hasida, H Mazo, K Choukri, S Goggi, J Mariani, A Moreno, N Calzolari, J Odijk & T Tokunaga (eds), LREC 2018 - 11th International Conference on Language Resources and Evaluation., 880, European Language Resources Association (ELRA), pp. 339-348, 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, 7/05/18. <http://www.lrec-conf.org/proceedings/lrec2018/summaries/880.html>

Systems' agreements and disagreements in temporal processing : An extensive error analysis of the TempEval-3 task. / Caselli, Tommaso; Morante, Roser.

LREC 2018 - 11th International Conference on Language Resources and Evaluation. ed. / Hitoshi Isahara; Bente Maegaard; Stelios Piperidis; Christopher Cieri; Thierry Declerck; Koiti Hasida; Helene Mazo; Khalid Choukri; Sara Goggi; Joseph Mariani; Asuncion Moreno; Nicoletta Calzolari; Jan Odijk; Takenobu Tokunaga. European Language Resources Association (ELRA), 2019. p. 339-348 880.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Systems' agreements and disagreements in temporal processing

T2 - 11th International Conference on Language Resources and Evaluation, LREC 2018

AU - Caselli, Tommaso

AU - Morante, Roser

PY - 2019

Y1 - 2019

N2 - In this article we review Temporal Processing systems that participated in the TempEval-3 task as a basis to develop our own system, that we also present and release. The system incorporates high level lexical semantic features, obtaining the best scores for event detection (F1-Class 72.24) and second best result for temporal relation classification from raw text (F1 29.69) when evaluated on the TempEval-3 data. Additionally, we analyse the errors of all TempEval-3 systems for which the output is publicly available with the purpose of finding out what are the weaknesses of current approaches. Although incorporating lexical semantics features increases the performance of our system, the error analysis shows that systems should incorporate inference mechanisms and world knowledge, as well as having strategies to compensate for data skewness.

AB - In this article we review Temporal Processing systems that participated in the TempEval-3 task as a basis to develop our own system, that we also present and release. The system incorporates high level lexical semantic features, obtaining the best scores for event detection (F1-Class 72.24) and second best result for temporal relation classification from raw text (F1 29.69) when evaluated on the TempEval-3 data. Additionally, we analyse the errors of all TempEval-3 systems for which the output is publicly available with the purpose of finding out what are the weaknesses of current approaches. Although incorporating lexical semantics features increases the performance of our system, the error analysis shows that systems should incorporate inference mechanisms and world knowledge, as well as having strategies to compensate for data skewness.

KW - Error analysis

KW - Temporal processing

KW - Written corpora

UR - http://www.scopus.com/inward/record.url?scp=85059907539&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85059907539&partnerID=8YFLogxK

UR - http://www.lrec-conf.org/proceedings/lrec2018/index.html

M3 - Conference contribution

AN - SCOPUS:85059907539

SP - 339

EP - 348

BT - LREC 2018 - 11th International Conference on Language Resources and Evaluation

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Piperidis, Stelios

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Hasida, Koiti

A2 - Mazo, Helene

A2 - Choukri, Khalid

A2 - Goggi, Sara

A2 - Mariani, Joseph

A2 - Moreno, Asuncion

A2 - Calzolari, Nicoletta

A2 - Odijk, Jan

A2 - Tokunaga, Takenobu

PB - European Language Resources Association (ELRA)

Y2 - 7 May 2018 through 12 May 2018

ER -

Caselli T, Morante R. Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task. In Isahara H, Maegaard B, Piperidis S, Cieri C, Declerck T, Hasida K, Mazo H, Choukri K, Goggi S, Mariani J, Moreno A, Calzolari N, Odijk J, Tokunaga T, editors, LREC 2018 - 11th International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA). 2019. p. 339-348. 880

Systems' agreements and disagreements in temporal processing: An extensive error analysis of the TempEval-3 task

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this