Preface: LM-KBC Challenge 2024

Jan Christoph Kalo*, Tuan Phong Nguyen, Simon Razniewski, Bohui Zhang

*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceedingForeword/postscriptAcademic

Abstract

Pretrained language models (LMs) have advanced a range of semantic tasks and have also shown promise for knowledge extraction from the models itself. Although several works have explored this ability in a setting called probing or prompting, the viability of knowledge base construction from LMs remains underexplored. In the 3rd edition of this challenge, participants were asked to build actual disambiguated knowledge bases from LMs, for given subjects and relations. In crucial difference to existing probing benchmarks like LAMA [1], we make no simplifying assumptions on relation cardinalities, i.e., a subject-entity can stand in relation with zero, one, or many object-entities. Furthermore, submissions need to go beyond just ranking predicted surface strings and materialize disambiguated entities in the output, which will be evaluated using established KB metrics of precision and recall. The challenge has a single track for LMs with a parameter level under 10-billion to fit into low computational requirements. The challenge received 8 submissions, of which 5 submitted a paper, and 4 were accepted for presentation. The challenge was collocated with a workshop on related topics, allowing to host extended discussions, related papers, and invited talks.

Original languageEnglish
Title of host publicationKBC-LM-LM-KBC 2024 Joint proceedings of the KBC-LM workshop and the LM-KBC challenge 2024
Subtitle of host publicationJoint proceedings of the 2nd workshop on Knowledge Base Construction from Pre-Trained Language Models (KBC-LM 2024) and the 3rd challenge on Language Models for Knowledge Base Construction (LM-KBC 2024) co-located with the 23nd International Semantic Web Conference (ISWC 2024) Baltimore, USA, November 12, 2024
EditorsSimon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jeff Z. Pan, Tuan-Phong Nguyen, Bohui Zhang
PublisherCEUR-WS.org
Pages1-5
Number of pages5
Publication statusPublished - 2024
EventJoint of the 2nd Workshop on Knowledge Base Construction from Pre-Trained Language Models and the 3rd Challenge on Language Models for Knowledge Base Construction, KBC-LM-LM-KBC 2024 - Baltimore, United States
Duration: 12 Nov 2024 → …

Publication series

NameCEUR Workshop Proceedings
PublisherCEUR Workshop Proceedings
Volume3853
ISSN (Print)1613-0073

Conference

ConferenceJoint of the 2nd Workshop on Knowledge Base Construction from Pre-Trained Language Models and the 3rd Challenge on Language Models for Knowledge Base Construction, KBC-LM-LM-KBC 2024
Country/TerritoryUnited States
CityBaltimore
Period12/11/24 → …

Bibliographical note

Publisher Copyright:
© 2024 Copyright for this paper by its authors.

Fingerprint

Dive into the research topics of 'Preface: LM-KBC Challenge 2024'. Together they form a unique fingerprint.

Cite this