Knowledge-based Entity Prediction for Improved Machine Perception in Autonomous Systems

Ruwan Wickramarachchi, Cory Henson, Amit ShetharXiv 2022 · DOI · arXiv

TL;DR

This paper introduces Knowledge-based Entity Prediction (KEP), a novel framework that uses structured relational knowledge to help autonomous systems "see" what their sensors might miss. By formalizing perception as a knowledge graph completion task, the authors demonstrate how machines can predict the presence of unobserved entities—like a child chasing a bouncing ball—using common-sense reasoning and external domain knowledge.

What Problem Does This Paper Address?

Current machine perception in autonomous systems (like self-driving cars or factory robots) relies heavily on deep learning and computer vision. While powerful, these systems are "brittle": they struggle with occlusions, sensor failures, and unexpected "edge cases."

The fundamental gap is a lack of contextual and common-sense reasoning. A standard vision model might detect a ball bouncing into the street, but it doesn't inherently know that a child is likely to follow. This paper addresses the "unrecognized entity" problem by moving beyond raw pixels to a higher level of semantic understanding.

Fig. 1. Knowledge-based Entity Prediction Prob- lem in AD

Methods

The authors propose representing the environment as a Scene Knowledge Graph (KG). In this graph, nodes represent entities (objects, events, locations) and edges represent their relationships (e.g., includes, participatesIn, straightAhead).

They formalize KEP as a knowledge completion task and propose three distinct algorithmic approaches:

  1. Link Prediction (LP): Using Knowledge Graph Embeddings (KGE) to represent nodes and relations in a low-dimensional latent space. The system predicts missing entities by calculating the likelihood of a specific "triple" (Scene, includesType, EntityType) being true.
  2. Association Rule Mining (ARM): A data-mining approach similar to market basket analysis. It identifies frequent patterns (e.g., "if a crosswalk and a pedestrian are present, the pedestrian is likely walking") to infer missing scene components.
  3. Collective Classification: A combinatorial optimization method that jointly determines labels for all nodes in a graph, leveraging the idea that knowing the type of one node helps identify its neighbors.
Fig. 3. Three possible solutions devised for KEP based on different machine learning/data mining techniques

The authors advocate for a cyclic perception process where the Scene KG is continuously updated with new sensor data and KEP predictions, creating a richer "world model" for the planning and control modules.

Fig. 2. Improved machine perception through KEP in Autonomous Driving

Datasets

The researchers evaluated their approach using the Pandaset autonomous driving dataset. They transformed this raw data into the Driving Scenes Knowledge Graph (DSKG), which is conformant to a specialized Driving Scenes Ontology (DSO). This allowed them to integrate real-world sensor observations with structured domain knowledge.

Key Insights and Ideas

  • Knowledge-Infused Learning: The paper argues that perception shouldn't just be about data; it should be about knowledge. By infusing deep learning with ontologies and common-sense graphs (like ConceptNet), systems can handle uncertainty more gracefully.
  • Psychological Parallel: The authors draw a fascinating parallel between KEP and Event Segmentation Theory (EST) in psychology, which suggests humans perceive events by constantly making predictions about the near future based on internal models of the world.
  • Explainability by Design: Unlike black-box neural networks, KEP provides a "paper trail." Because it uses explicit relations (e.g., a pedestrian is predicted because of a nearby school zone and a detected ball), the system’s reasoning is inherently more interpretable for safety audits and accident investigations.

Key Results

The authors tested three KGE algorithms (TransE, HolE, and ConvKB) on the DSKG-PR dataset. The HolE (Holographic Embeddings) algorithm was the clear winner, achieving:

  • 88.91% Accuracy in predicting missing entities.
  • 0.87 Hits@1, meaning the correct missing entity was the top-ranked prediction 87% of the time.
  • 0.93 Mean Reciprocal Rank (MRR), indicating very high confidence in its correct predictions.

These results suggest that relational knowledge is a highly effective signal for filling in the blanks left by traditional computer vision.

Limitations and Future Work

While promising, the paper notes several areas for growth:

  • Computational Complexity: Some methods, like ConvKB, are computationally expensive in terms of space and time, which could be a hurdle for real-time deployment in fast-moving vehicles.
  • Real-time Integration: The current study focuses on "offline" evaluation. The next step is integrating KEP into the active control loops of real autonomous systems to see how it affects actual driving behavior.
  • Scalability: Expanding the Scene KG to include even more diverse external knowledge (legal rules, ethical constraints, hyper-local weather) remains a significant engineering challenge.

Summary updated

Abstract

Knowledge-based entity prediction (KEP) is a novel task that aims to improve machine perception in autonomous systems. KEP leverages relational knowledge from heterogeneous sources in predicting potentially unrecognized entities. In this paper, we provide a formal definition of KEP as a knowledge completion task. Three potential solutions are then introduced, which employ several machine learning and data mining techniques. Finally, the applicability of KEP is demonstrated on two autonomous systems from different domains; namely, autonomous driving and smart manufacturing. We argue that in complex real-world systems, the use of KEP would significantly improve machine perception while pushing the current technology one step closer to achieving full autonomy.
BibTeX
@misc{wickramarachchi2022knowledge,
  author = {Ruwan Wickramarachchi and Cory Henson and Amit Sheth},
  title = {{Knowledge-based Entity Prediction for Improved Machine Perception in Autonomous Systems}},
  year = {2022},
  eprint = {2203.16616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.16616},
}
Figure