TL;DR
This paper introduces a novel approach to autonomous vehicle motion prediction by modeling traffic environments as spatial semantic scene graphs. By using Graph Neural Networks (GNNs) to explicitly capture relationships like "following" or "intersecting" between vehicles, the proposed method improves acceleration prediction by up to 12% over edge-agnostic baselines, and up to 73% when integrated with temporal history via LSTMs.
What Problem Does This Paper Address?
Safe autonomous driving requires a vehicle to accurately anticipate the future actions of surrounding traffic participants. While human drivers naturally interpret scenes through the lens of traffic rules and social interactions (e.g., "that car is cutting into my lane, so I should brake"), most deep learning models struggle to capture these explicit relationships.
Current state-of-the-art motion predictors typically rely on raw sensor data, grid-based representations, or rasterized bird's-eye-view images. These methods force neural networks to implicitly learn the physics and geometry of the road from scratch. This paper addresses this gap by explicitly structuring the driving context using semantic relations based on road topology, making it easier for models to reason about vehicle interactions.
Methods
The authors propose a framework that transforms a raw traffic scene into a structured graph representation and processes it using GNNs.
- Semantic Scene Model (SSM): The traffic scene is converted into a directed graph $G = (V, E)$.
- Nodes ($V$): Represent traffic participants, containing features like vehicle classification and velocity.
- Edges ($E$): Represent spatial semantic relationships, categorized into three types: longitudinal (vehicles in the same lane), lateral (vehicles in adjacent lanes), and intersecting (vehicles on paths that will merge or cross).
- Edge Attributes: Calculated using Frenet coordinates (road-aligned coordinates), edges store the precise distance along the roadway between participants and an assignment probability based on lane alignment.
- Network Architectures:
- Single Step Network: Uses a Graph Convolution (GConv) layer with a message-passing mechanism. Node states are updated based on their neighbors' features and the connecting edge attributes. A Multi-Layer Perceptron (MLP) then predicts the immediate acceleration ($a_i$) for each vehicle.
- Recurrent Network: To capture how a scene evolves over time, the authors chain multiple GConv layers across a temporal sequence of frames ($T$ previous scenes) and feed the resulting structural embeddings into a Long Short-Term Memory (LSTM) network.
Datasets
The models were trained and evaluated on two distinct, publicly available datasets:
- INTERACTION Dataset: Features drone-recorded tactical driving behavior at high-density intersections and roundabouts, complete with highly accurate HD maps.
- PandaSet: Comprises short (8-second) urban driving sequences captured from a vehicle equipped with cameras and LiDAR.
Key Insights and Ideas
- Edges Matter: An ablation study ("Single Step no edge data") revealed that simply knowing who is in the scene isn't enough. Explicitly incorporating how vehicles relate to one another via semantic edges provides a strong inductive bias that significantly boosts prediction accuracy.
- Frenet Space Simplifies Geometry: By calculating distances along the centerlines of lanes rather than using raw Cartesian coordinates, the model remains robust to lateral shifts and complex road geometries like merging lanes or roundabouts.
- History is Crucial: Moving from a single-frame snapshot to a temporal sequence (using 5 to 15 historical frames) yielded the largest performance leap, demonstrating that interaction dynamics are fundamentally historical.
Key Results
The authors evaluated performance using Linear Error (L1-loss) and Squared Error (MSE-loss) against two simple baselines: Baseline Mean (always outputs the dataset's average acceleration) and Baseline Zero (always outputs zero acceleration).
- Relational Benefit: The Single Step model outperformed the edge-free ablation model by 12%, proving the value of explicit spatial-semantic edges.
- Temporal Boost: The Recurrent15 model (using 15 historical frames) achieved a 70% to 73% improvement over the baselines on the INTERACTION dataset, dropping the L1 acceleration error down to 0.170.
- Dataset Characteristics: The model performed better on the INTERACTION dataset than on PandaSet. The authors attribute this to the fact that PandaSet scenes are noisier, containing roughly 24 participants per scene with 12 relations each, compared to INTERACTION's cleaner average of 5 vehicles and 2 relations per scene.
| Model | Dataset | L1 Loss | MSE Loss |
|---|---|---|---|
| Single Step | INTERACTION | 0.494 | 0.451 |
| Single Step (no edges) | INTERACTION | 0.552 | 0.538 |
| Recurrent15 | INTERACTION | 0.170 | 0.085 |
| Baseline Zero | INTERACTION | 0.599 | 0.622 |
Limitations and Future Work
While promising, the approach currently has a few notable limitations:
- Longitudinal Only: The model only predicts acceleration/deceleration (longitudinal control) and completely ignores steering (lateral control).
- Missing Context: It does not yet incorporate static scene elements like road curvature, traffic lights, or stop lines directly into the graph.
- Simplified Traffic: Vulnerable road users like pedestrians and cyclists are not yet integrated into the semantic relation framework.
The authors plan to expand the graph schema to include these missing elements and explore how these structured scene graph embeddings can be fused with state-of-the-art image-based trajectory predictors to create more robust, hybrid models.
Summary updated