Machine Learning on Multimodal Knowledge Graphs: Opportunities, Challenges, and Methods for Learning on Real-World Heterogeneous and Spatially-Oriented Knowledge

The knowledge graph is a data model in which knowledge, information, and data are all encoded in graph form using the same basic building blocks. This knowledge can be entirely made up of objects, expressing all information through their connectivity, but knowledge graphs are also capable of seamlessly integrating other forms of information, including images, natural language, and spatial information, making the knowledge graph a suitable choice to model heterogeneous knowledge with. With a wealth of heterogeneous knowledge already available in knowledge graph format, and with the expectation that this amount is only to grow in the future, the knowledge graph data model becomes ever more interesting for machine learning scientists and practitioners to learn on. This thesis identifies the most essential opportunities and challenges that arise with machine learning on heterogeneous knowledge, encoded as knowledge graph, and investigates 1) how machine learning models can be build that incorporate this heterogeneity and to what extent this affects their performance, and 2) how data scientists can use such models to discover interesting patterns in knowledge graphs that may help experts perform various downstream tasks. These lines are addressed in six chapters and along three dimensions. These dimensions concern to what extent 1) contextual and 2) multimodal information are included in the learning process, and 3) the level of involvement of experts in this process. Several reusable scientific resources were created during the investigation of the aforementioned topics. These resources include a) an ontology for binary-encoded data, 2) a collection of multimodal benchmark datasets for machine learning on knowledge graphs, and 3) a multimodal message-passing model, called the MR-GCN, which can consume any arbitrary knowledge graph out of the box. With the adoption of knowledge graphs by organisations around the world, the interest in machine learning on this data model has increased considerately. The next logical step in this field is the merging of statistical and logical approaches, by developing machine learning models that are designed with neuro- symbolic learning from the outset, and which are just as capable at learning on the data in a graph as that they are at learning on its semantics. By developing neuro-symbolic models, data scientists are paving the road towards true end-to-end machine learning on knowledge graphs.