Using linked data to help robots understand product-related actions

Household robots need semantics to understand that a detergent is a cleaning product that can be used to clean physical objects like a table, but laundry detergent is only used to clean/wash laundry. A safely acting autonomous robot should also know that both will not be used as ingredients for meal preparation. We propose a new approach to connect robot sensor data to Linked Data in order to give robotic agents semantic product information about objects that can be found in their environment so that the action to be performed with a given object can be inferred. For this, we use the robot’s belief state when recognizing a product and link it to a product ontology that follows Semantic Web standards. We then use the product class information to fetch further information from external sources like Wikidata or ConceptNet that contain action information (e.g. laundry detergent is used for laundering). At last, the action results are mapped to internally known actions of the robotic agent so that it knows which action can be performed with the perceived object.