An Evaluation of Knowledge Graph Embeddings for Autonomous Driving Data: Experience and Practice

Ruwan Wickramarachchi, Cory Henson, Amit ShetharXiv 2020 · arXiv

TL;DR

This paper presents the first systematic evaluation of Knowledge Graph Embeddings (KGEs) for autonomous driving data, exploring how structural graph detail impacts embedding quality. The authors demonstrate that adding explicit semantic paths and inferred types significantly boosts embedding performance, with translational algorithms like TransE outperforming semantic matching models.

What Problem Does This Paper Address?

Autonomous driving (AD) systems generate massive amounts of heterogeneous data from cameras, LiDAR, and RADAR. To achieve true "scene understanding," vehicles need to connect these raw sensor inputs with high-level concepts like objects, events, time, and location.

Knowledge Graphs (KGs) are excellent for organizing this domain knowledge, but machine learning models cannot easily ingest symbolic graphs directly. Enter Knowledge Graph Embeddings (KGEs)—a technique that maps graphs into continuous vector spaces, facilitating "neuro-symbolic fusion" (combining symbolic logic with deep learning). However, it remains unclear how the level of informational detail in a KG affects the quality of these embeddings, and which embedding algorithms work best for complex driving scenes.

Methods

The authors built a "Scene Ontology" to formally define driving situations, sub-scenes, and features of interest (FoIs).

Figure 1: Formal definition of a scene, as defined by the Scene Ontology

Using this ontology, they generated three versions of KGs with increasing levels of informational detail:

  1. Base KG: Standard scene-to-sample-to-object triples.
  2. KG with Inferred Types: Using a semantic reasoner to make implicit hierarchical relationships explicit (e.g., explicitly stating that a Car instance is also a Vehicle and a FeatureOfInterest).
  3. KG with Include Paths: Adding direct "shortcut" relations between a high-level scene and the objects detected within its sub-scenes.
Figure 3: Example KGs: (a) Base KG, (b) KG w/ inferred types, and (c) KG w/ include path

They trained three popular embedding algorithms representing different paradigms: TransE (distance-based), RESCAL, and HolE (semantic matching-based). Finally, they systematically evaluated the embeddings across four dimensions using metrics like categorization measure, coherence measure, and semantic transition distance.

Figure 6: Four dimensions involved in our evaluation

Datasets

The authors evaluated their approach using two popular benchmark autonomous driving datasets:

  • NuScenes (by Aptiv): Contains 850 driving segments (20-second clips) with 34,149 samples, annotated with 23 object/event categories.
  • Lyft-Level5: Contains 180 driving segments with 22,680 samples, annotated with 9 object/event categories.

Key Insights and Ideas

  • More Detail = Better Embeddings: Explicitly adding inferred types and multi-hop paths doesn't change the logical expressivity of the graph, but it drastically improves the geometric quality of the learned vector space.
  • TransE Rules the Road: The translational distance-based algorithm (TransE) consistently outperformed semantic matching models (RESCAL, HolE). Distance-based scoring functions appear much better suited for preserving hierarchical type and relational semantics in spatial-temporal domains.
  • Standard Metrics Can Fail: The standard "coherence measure" (which checks if the nearest neighbors of an entity share a background concept) proved unsuitable for this domain. Because embeddings naturally cluster by specific scenes or co-occurring events, a top-level concept like "Human" will have a neighborhood filled with diverse, scene-specific entities, leading to misleadingly low coherence scores.

Key Results

  • Scalability Bottlenecks: RESCAL failed to scale to the NuScenes dataset, which contained over 10.8 million triples and 2.1 million entities, highlighting a major limitation for production-scale AD data.
  • Clear Semantic Clustering: t-SNE visualizations showed clear semantic clustering. For instance, instances of events like stopped car, moving car, and parked car tightly clustered around the Car feature of interest.
Figure 4: Clustering of FoIs with events in NuScenes dataset
  • Relational Quality: Adding explicit "include paths" drastically improved the semantic transition distance for the isPartOf relation across all algorithms, proving that structural shortcuts help models capture scene-subscene relationships.

Limitations and Future Work

  • Algorithmic Limitations: Semantic matching models like RESCAL are too computationally expensive for massive AD datasets, necessitating more space-efficient alternatives like HolE or TransE.
  • Manual Pre-processing: Currently, the richer informational details (inferred types and paths) must be injected via a reasoner before embedding. The authors suggest that future KGE algorithms should automatically extract these latent semantics during training.
  • Downstream Validation: While this paper focuses on intrinsic evaluation (the geometric quality of the embeddings), the authors note that future work must validate these KGEs on extrinsic downstream tasks like scene similarity search, anomaly detection, or vehicle trajectory prediction.

Summary updated

Abstract

The autonomous driving (AD) industry is exploring the use of knowledge graphs (KGs) to manage the vast amount of heterogeneous data generated from vehicular sensors. The various types of equipped sensors include video, LIDAR and RADAR. Scene understanding is an important topic in AD which requires consideration of various aspects of a scene, such as detected objects, events, time and location. Recent work on knowledge graph embeddings (KGEs) - an approach that facilitates neuro-symbolic fusion - has shown to improve the predictive performance of machine learning models. With the expectation that neuro-symbolic fusion through KGEs will improve scene understanding, this research explores the generation and evaluation of KGEs for autonomous driving data. We also present an investigation of the relationship between the level of informational detail in a KG and the quality of its derivative embeddings. By systematically evaluating KGEs along four dimensions -- i.e. quality metrics, KG informational detail, algorithms, and datasets -- we show that (1) higher levels of informational detail in KGs lead to higher quality embeddings, (2) type and relation semantics are better captured by the semantic transitional distance-based TransE algorithm, and (3) some metrics, such as coherence measure, may not be suitable for intrinsically evaluating KGEs in this domain. Additionally, we also present an (early) investigation of the usefulness of KGEs for two use-cases in the AD domain.
BibTeX
@misc{wickramarachchi2020evaluation,
  author = {Ruwan Wickramarachchi and Cory Henson and Amit Sheth},
  title = {{An Evaluation of Knowledge Graph Embeddings for Autonomous Driving Data: Experience and Practice}},
  year = {2020},
  eprint = {2003.00344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.00344},
}
Figure