kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning

Peter Bloem; Xander Wilcke; Lucas van Berkel; Victor de Boer

doi:10.1007/978-3-030-77385-4_37

kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning

Peter Bloem^*, Xander Wilcke, Lucas van Berkel, Victor de Boer

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

749 Downloads (Pure)

Abstract

Graph neural networks and other machine learning models offer a promising direction for machine learning on relational and multimodal data. Until now, however, progress in this area is difficult to gauge. This is primarily due to a limited number of datasets with (a) a high enough number of labeled nodes in the test set for precise measurement of performance, and (b) a rich enough variety of multimodal information to learn from. We introduce a set of new benchmark tasks for node classification on RDF-encoded knowledge graphs. We focus primarily on node classification, since this setting cannot be solved purely by node embedding models. For each dataset, we provide test and validation sets of at least 1000 instances, with some over 10000. Each task can be performed in a purely relational manner, or with multimodal information. All datasets are packaged in a CSV format that is easily consumable in any machine learning environment, together with the original source data in RDF and pre-processing code for full provenance. We provide code for loading the data into numpy and pytorch. We compute performance for several baseline models.

Original language	English
Title of host publication	The Semantic Web
Subtitle of host publication	18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings
Editors	Ruben Verborgh, Katja Hose, Heiko Paulheim, Pierre-Antoine Champin, Maria Maleshkova, Oscar Corcho, Petar Ristoski, Mehwish Alam
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	614-630
Number of pages	17
ISBN (Electronic)	9783030773854
ISBN (Print)	9783030773847
DOIs	https://doi.org/10.1007/978-3-030-77385-4_37
Publication status	Published - 2021
Event	18th European Semantic Web Conference, ESWC 2021 - Virtual, Online Duration: 6 Jun 2021 → 10 Jun 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12731 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th European Semantic Web Conference, ESWC 2021
City	Virtual, Online
Period	6/06/21 → 10/06/21

Bibliographical note

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.

Keywords

Knowledge graphs
Machine learning
Message passing models
Multimodal learning

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-77385-4_37

Bloem2021_Chapter_KgbenchACollectionOfKnowledgeGraphDatasetsForFinal published version, 365 KBLicence: Article 25fa Dutch Copyright Act

Persistent URL (handle)

Persistent link

Cite this

Bloem, P., Wilcke, X., van Berkel, L., & de Boer, V. (2021). kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning. In R. Verborgh, K. Hose, H. Paulheim, P.-A. Champin, M. Maleshkova, O. Corcho, P. Ristoski, & M. Alam (Eds.), The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings (pp. 614-630). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12731 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-77385-4_37

Bloem, Peter ; Wilcke, Xander ; van Berkel, Lucas et al. / kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning. The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings. editor / Ruben Verborgh ; Katja Hose ; Heiko Paulheim ; Pierre-Antoine Champin ; Maria Maleshkova ; Oscar Corcho ; Petar Ristoski ; Mehwish Alam. Springer Science and Business Media Deutschland GmbH, 2021. pp. 614-630 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9dfa7a0d6b51447da453b39c8d8768ae,

title = "kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning",

abstract = "Graph neural networks and other machine learning models offer a promising direction for machine learning on relational and multimodal data. Until now, however, progress in this area is difficult to gauge. This is primarily due to a limited number of datasets with (a) a high enough number of labeled nodes in the test set for precise measurement of performance, and (b) a rich enough variety of multimodal information to learn from. We introduce a set of new benchmark tasks for node classification on RDF-encoded knowledge graphs. We focus primarily on node classification, since this setting cannot be solved purely by node embedding models. For each dataset, we provide test and validation sets of at least 1000 instances, with some over 10000. Each task can be performed in a purely relational manner, or with multimodal information. All datasets are packaged in a CSV format that is easily consumable in any machine learning environment, together with the original source data in RDF and pre-processing code for full provenance. We provide code for loading the data into numpy and pytorch. We compute performance for several baseline models.",

keywords = "Knowledge graphs, Machine learning, Message passing models, Multimodal learning",

author = "Peter Bloem and Xander Wilcke and {van Berkel}, Lucas and {de Boer}, Victor",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; 18th European Semantic Web Conference, ESWC 2021 ; Conference date: 06-06-2021 Through 10-06-2021",

year = "2021",

doi = "10.1007/978-3-030-77385-4_37",

language = "English",

isbn = "9783030773847",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "614--630",

editor = "Ruben Verborgh and Katja Hose and Heiko Paulheim and Pierre-Antoine Champin and Maria Maleshkova and Oscar Corcho and Petar Ristoski and Mehwish Alam",

booktitle = "The Semantic Web",

address = "Germany",

}

Bloem, P , Wilcke, X, van Berkel, L & de Boer, V 2021, kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning. in R Verborgh, K Hose, H Paulheim, P-A Champin, M Maleshkova, O Corcho, P Ristoski & M Alam (eds), The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12731 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 614-630, 18th European Semantic Web Conference, ESWC 2021, Virtual, Online, 6/06/21. https://doi.org/10.1007/978-3-030-77385-4_37

kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning. / Bloem, Peter ; Wilcke, Xander; van Berkel, Lucas et al.
The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings. ed. / Ruben Verborgh; Katja Hose; Heiko Paulheim; Pierre-Antoine Champin; Maria Maleshkova; Oscar Corcho; Petar Ristoski; Mehwish Alam. Springer Science and Business Media Deutschland GmbH, 2021. p. 614-630 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12731 LNCS).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning

AU - Bloem, Peter

AU - Wilcke, Xander

AU - van Berkel, Lucas

AU - de Boer, Victor

PY - 2021

Y1 - 2021

N2 - Graph neural networks and other machine learning models offer a promising direction for machine learning on relational and multimodal data. Until now, however, progress in this area is difficult to gauge. This is primarily due to a limited number of datasets with (a) a high enough number of labeled nodes in the test set for precise measurement of performance, and (b) a rich enough variety of multimodal information to learn from. We introduce a set of new benchmark tasks for node classification on RDF-encoded knowledge graphs. We focus primarily on node classification, since this setting cannot be solved purely by node embedding models. For each dataset, we provide test and validation sets of at least 1000 instances, with some over 10000. Each task can be performed in a purely relational manner, or with multimodal information. All datasets are packaged in a CSV format that is easily consumable in any machine learning environment, together with the original source data in RDF and pre-processing code for full provenance. We provide code for loading the data into numpy and pytorch. We compute performance for several baseline models.

AB - Graph neural networks and other machine learning models offer a promising direction for machine learning on relational and multimodal data. Until now, however, progress in this area is difficult to gauge. This is primarily due to a limited number of datasets with (a) a high enough number of labeled nodes in the test set for precise measurement of performance, and (b) a rich enough variety of multimodal information to learn from. We introduce a set of new benchmark tasks for node classification on RDF-encoded knowledge graphs. We focus primarily on node classification, since this setting cannot be solved purely by node embedding models. For each dataset, we provide test and validation sets of at least 1000 instances, with some over 10000. Each task can be performed in a purely relational manner, or with multimodal information. All datasets are packaged in a CSV format that is easily consumable in any machine learning environment, together with the original source data in RDF and pre-processing code for full provenance. We provide code for loading the data into numpy and pytorch. We compute performance for several baseline models.

KW - Knowledge graphs

KW - Machine learning

KW - Message passing models

KW - Multimodal learning

UR - http://www.scopus.com/inward/record.url?scp=85111134694&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85111134694&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-77385-4_37

DO - 10.1007/978-3-030-77385-4_37

M3 - Conference contribution

AN - SCOPUS:85111134694

SN - 9783030773847

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 614

EP - 630

BT - The Semantic Web

A2 - Verborgh, Ruben

A2 - Hose, Katja

A2 - Paulheim, Heiko

A2 - Champin, Pierre-Antoine

A2 - Maleshkova, Maria

A2 - Corcho, Oscar

A2 - Ristoski, Petar

A2 - Alam, Mehwish

PB - Springer Science and Business Media Deutschland GmbH

T2 - 18th European Semantic Web Conference, ESWC 2021

Y2 - 6 June 2021 through 10 June 2021

ER -

Bloem P , Wilcke X, van Berkel L, de Boer V. kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning. In Verborgh R, Hose K, Paulheim H, Champin PA, Maleshkova M, Corcho O, Ristoski P, Alam M, editors, The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 614-630. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2021 May 31. doi: 10.1007/978-3-030-77385-4_37

kgbench: A Collection of Knowledge Graph Datasets for Evaluating Relational and Multimodal Machine Learning

Abstract

Publication series

Conference

Bibliographical note

Keywords

UN SDGs

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this