TL;DR
This paper presents the first systematic evaluation of Knowledge Graph Embeddings (KGEs) for autonomous driving data, exploring how structural graph detail impacts embedding quality. The authors demonstrate that adding explicit semantic paths and inferred types significantly boosts embedding performance, with translational algorithms like TransE outperforming semantic matching models.
What Problem Does This Paper Address?
Autonomous driving (AD) systems generate massive amounts of heterogeneous data from cameras, LiDAR, and RADAR. To achieve true "scene understanding," vehicles need to connect these raw sensor inputs with high-level concepts like objects, events, time, and location.
Knowledge Graphs (KGs) are excellent for organizing this domain knowledge, but machine learning models cannot easily ingest symbolic graphs directly. Enter Knowledge Graph Embeddings (KGEs)—a technique that maps graphs into continuous vector spaces, facilitating "neuro-symbolic fusion" (combining symbolic logic with deep learning). However, it remains unclear how the level of informational detail in a KG affects the quality of these embeddings, and which embedding algorithms work best for complex driving scenes.
Methods
The authors built a "Scene Ontology" to formally define driving situations, sub-scenes, and features of interest (FoIs).
Using this ontology, they generated three versions of KGs with increasing levels of informational detail:
- Base KG: Standard scene-to-sample-to-object triples.
- KG with Inferred Types: Using a semantic reasoner to make implicit hierarchical relationships explicit (e.g., explicitly stating that a
Carinstance is also aVehicleand aFeatureOfInterest). - KG with Include Paths: Adding direct "shortcut" relations between a high-level scene and the objects detected within its sub-scenes.
They trained three popular embedding algorithms representing different paradigms: TransE (distance-based), RESCAL, and HolE (semantic matching-based). Finally, they systematically evaluated the embeddings across four dimensions using metrics like categorization measure, coherence measure, and semantic transition distance.
Datasets
The authors evaluated their approach using two popular benchmark autonomous driving datasets:
- NuScenes (by Aptiv): Contains 850 driving segments (20-second clips) with 34,149 samples, annotated with 23 object/event categories.
- Lyft-Level5: Contains 180 driving segments with 22,680 samples, annotated with 9 object/event categories.
Key Insights and Ideas
- More Detail = Better Embeddings: Explicitly adding inferred types and multi-hop paths doesn't change the logical expressivity of the graph, but it drastically improves the geometric quality of the learned vector space.
- TransE Rules the Road: The translational distance-based algorithm (TransE) consistently outperformed semantic matching models (RESCAL, HolE). Distance-based scoring functions appear much better suited for preserving hierarchical type and relational semantics in spatial-temporal domains.
- Standard Metrics Can Fail: The standard "coherence measure" (which checks if the nearest neighbors of an entity share a background concept) proved unsuitable for this domain. Because embeddings naturally cluster by specific scenes or co-occurring events, a top-level concept like "Human" will have a neighborhood filled with diverse, scene-specific entities, leading to misleadingly low coherence scores.
Key Results
- Scalability Bottlenecks: RESCAL failed to scale to the NuScenes dataset, which contained over 10.8 million triples and 2.1 million entities, highlighting a major limitation for production-scale AD data.
- Clear Semantic Clustering: t-SNE visualizations showed clear semantic clustering. For instance, instances of events like
stopped car,moving car, andparked cartightly clustered around theCarfeature of interest.
- Relational Quality: Adding explicit "include paths" drastically improved the semantic transition distance for the
isPartOfrelation across all algorithms, proving that structural shortcuts help models capture scene-subscene relationships.
Limitations and Future Work
- Algorithmic Limitations: Semantic matching models like RESCAL are too computationally expensive for massive AD datasets, necessitating more space-efficient alternatives like HolE or TransE.
- Manual Pre-processing: Currently, the richer informational details (inferred types and paths) must be injected via a reasoner before embedding. The authors suggest that future KGE algorithms should automatically extract these latent semantics during training.
- Downstream Validation: While this paper focuses on intrinsic evaluation (the geometric quality of the embeddings), the authors note that future work must validate these KGEs on extrinsic downstream tasks like scene similarity search, anomaly detection, or vehicle trajectory prediction.
Summary updated