Semantic overfitting: what `world' do we consider when evaluating disambiguation of text?

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Semantic text processing faces the challenge of defining the relation between lexical expressions
and the world to which they make reference within a period of time. It is unclear whether the
current test sets used to evaluate disambiguation tasks are representative for the full complexity
considering this time-anchored relation, resulting in semantic overfitting to a specific period and
the frequent phenomena within. We conceptualize and formalize a set of metrics which eval-
uate this complexity of datasets. We provide evidence for their applicability on five different
disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose
a time-based, metric-aware method for developing datasets in a systematic and semi-automated
manner, as well as an event-based QA task
LanguageEnglish
Title of host publicationProceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Pages1180-1191
Number of pages12
Publication statusPublished - 2016

Fingerprint

Semantics
Text processing

Cite this

Ilievski, F., Postma, M. C., & Vossen, P. T. J. M. (2016). Semantic overfitting: what `world' do we consider when evaluating disambiguation of text? In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 1180-1191)
Ilievski, F. ; Postma, M.C. ; Vossen, P.T.J.M. / Semantic overfitting: what `world' do we consider when evaluating disambiguation of text?. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers . 2016. pp. 1180-1191
@inproceedings{36c60d519a6443c3a469f4f13bb13f8c,
title = "Semantic overfitting: what `world' do we consider when evaluating disambiguation of text?",
abstract = "Semantic text processing faces the challenge of defining the relation between lexical expressionsand the world to which they make reference within a period of time. It is unclear whether thecurrent test sets used to evaluate disambiguation tasks are representative for the full complexityconsidering this time-anchored relation, resulting in semantic overfitting to a specific period andthe frequent phenomena within. We conceptualize and formalize a set of metrics which eval-uate this complexity of datasets. We provide evidence for their applicability on five differentdisambiguation tasks. To challenge semantic overfitting of disambiguation systems, we proposea time-based, metric-aware method for developing datasets in a systematic and semi-automatedmanner, as well as an event-based QA task",
author = "F. Ilievski and M.C. Postma and P.T.J.M. Vossen",
year = "2016",
language = "English",
pages = "1180--1191",
booktitle = "Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers",

}

Ilievski, F, Postma, MC & Vossen, PTJM 2016, Semantic overfitting: what `world' do we consider when evaluating disambiguation of text? in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers . pp. 1180-1191.

Semantic overfitting: what `world' do we consider when evaluating disambiguation of text? / Ilievski, F.; Postma, M.C.; Vossen, P.T.J.M.

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers . 2016. p. 1180-1191.

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Semantic overfitting: what `world' do we consider when evaluating disambiguation of text?

AU - Ilievski, F.

AU - Postma, M.C.

AU - Vossen, P.T.J.M.

PY - 2016

Y1 - 2016

N2 - Semantic text processing faces the challenge of defining the relation between lexical expressionsand the world to which they make reference within a period of time. It is unclear whether thecurrent test sets used to evaluate disambiguation tasks are representative for the full complexityconsidering this time-anchored relation, resulting in semantic overfitting to a specific period andthe frequent phenomena within. We conceptualize and formalize a set of metrics which eval-uate this complexity of datasets. We provide evidence for their applicability on five differentdisambiguation tasks. To challenge semantic overfitting of disambiguation systems, we proposea time-based, metric-aware method for developing datasets in a systematic and semi-automatedmanner, as well as an event-based QA task

AB - Semantic text processing faces the challenge of defining the relation between lexical expressionsand the world to which they make reference within a period of time. It is unclear whether thecurrent test sets used to evaluate disambiguation tasks are representative for the full complexityconsidering this time-anchored relation, resulting in semantic overfitting to a specific period andthe frequent phenomena within. We conceptualize and formalize a set of metrics which eval-uate this complexity of datasets. We provide evidence for their applicability on five differentdisambiguation tasks. To challenge semantic overfitting of disambiguation systems, we proposea time-based, metric-aware method for developing datasets in a systematic and semi-automatedmanner, as well as an event-based QA task

UR - https://www.researchgate.net/publication/311774976_Semantic_overfitting_what_world'_do_we_consider_when_evaluating_disambiguation_of_text

M3 - Conference contribution

SP - 1180

EP - 1191

BT - Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers

ER -

Ilievski F, Postma MC, Vossen PTJM. Semantic overfitting: what `world' do we consider when evaluating disambiguation of text? In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers . 2016. p. 1180-1191