Abstract
Semantic text processing faces the challenge of defining the relation between lexical expressions
and the world to which they make reference within a period of time. It is unclear whether the
current test sets used to evaluate disambiguation tasks are representative for the full complexity
considering this time-anchored relation, resulting in semantic overfitting to a specific period and
the frequent phenomena within. We conceptualize and formalize a set of metrics which eval-
uate this complexity of datasets. We provide evidence for their applicability on five different
disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose
a time-based, metric-aware method for developing datasets in a systematic and semi-automated
manner, as well as an event-based QA task
and the world to which they make reference within a period of time. It is unclear whether the
current test sets used to evaluate disambiguation tasks are representative for the full complexity
considering this time-anchored relation, resulting in semantic overfitting to a specific period and
the frequent phenomena within. We conceptualize and formalize a set of metrics which eval-
uate this complexity of datasets. We provide evidence for their applicability on five different
disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose
a time-based, metric-aware method for developing datasets in a systematic and semi-automated
manner, as well as an event-based QA task
| Original language | English |
|---|---|
| Title of host publication | Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers |
| Pages | 1180-1191 |
| Number of pages | 12 |
| Publication status | Published - 2016 |
