Relation-based Motion Prediction using Traffic Scene Graphs

Maximilian Zipfl, Felix Hertlein, Achim Rettinger, Steffen Thoma, Lavdim Halilaj, Juergen Luettin, Stefan Schmid, Cory HensonarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a novel approach to autonomous vehicle motion prediction by modeling traffic environments as spatial semantic scene graphs. By using Graph Neural Networks (GNNs) to explicitly capture relationships like "following" or "intersecting" between vehicles, the proposed method improves acceleration prediction by up to 12% over edge-agnostic baselines, and up to 73% when integrated with temporal history via LSTMs.

What Problem Does This Paper Address?

Safe autonomous driving requires a vehicle to accurately anticipate the future actions of surrounding traffic participants. While human drivers naturally interpret scenes through the lens of traffic rules and social interactions (e.g., "that car is cutting into my lane, so I should brake"), most deep learning models struggle to capture these explicit relationships.

Current state-of-the-art motion predictors typically rely on raw sensor data, grid-based representations, or rasterized bird's-eye-view images. These methods force neural networks to implicitly learn the physics and geometry of the road from scratch. This paper addresses this gap by explicitly structuring the driving context using semantic relations based on road topology, making it easier for models to reason about vehicle interactions.

Methods

The authors propose a framework that transforms a raw traffic scene into a structured graph representation and processes it using GNNs.

  1. Semantic Scene Model (SSM): The traffic scene is converted into a directed graph $G = (V, E)$.
  2. Nodes ($V$): Represent traffic participants, containing features like vehicle classification and velocity.
  3. Edges ($E$): Represent spatial semantic relationships, categorized into three types: longitudinal (vehicles in the same lane), lateral (vehicles in adjacent lanes), and intersecting (vehicles on paths that will merge or cross).
  4. Edge Attributes: Calculated using Frenet coordinates (road-aligned coordinates), edges store the precise distance along the roadway between participants and an assignment probability based on lane alignment.
Fig. 1: Traffic scene seen from bird’s eye view.
  1. Network Architectures:
  2. Single Step Network: Uses a Graph Convolution (GConv) layer with a message-passing mechanism. Node states are updated based on their neighbors' features and the connecting edge attributes. A Multi-Layer Perceptron (MLP) then predicts the immediate acceleration ($a_i$) for each vehicle.
  3. Recurrent Network: To capture how a scene evolves over time, the authors chain multiple GConv layers across a temporal sequence of frames ($T$ previous scenes) and feed the resulting structural embeddings into a Long Short-Term Memory (LSTM) network.

Datasets

The models were trained and evaluated on two distinct, publicly available datasets:

  • INTERACTION Dataset: Features drone-recorded tactical driving behavior at high-density intersections and roundabouts, complete with highly accurate HD maps.
  • PandaSet: Comprises short (8-second) urban driving sequences captured from a vehicle equipped with cameras and LiDAR.

Key Insights and Ideas

  • Edges Matter: An ablation study ("Single Step no edge data") revealed that simply knowing who is in the scene isn't enough. Explicitly incorporating how vehicles relate to one another via semantic edges provides a strong inductive bias that significantly boosts prediction accuracy.
  • Frenet Space Simplifies Geometry: By calculating distances along the centerlines of lanes rather than using raw Cartesian coordinates, the model remains robust to lateral shifts and complex road geometries like merging lanes or roundabouts.
  • History is Crucial: Moving from a single-frame snapshot to a temporal sequence (using 5 to 15 historical frames) yielded the largest performance leap, demonstrating that interaction dynamics are fundamentally historical.

Key Results

The authors evaluated performance using Linear Error (L1-loss) and Squared Error (MSE-loss) against two simple baselines: Baseline Mean (always outputs the dataset's average acceleration) and Baseline Zero (always outputs zero acceleration).

  • Relational Benefit: The Single Step model outperformed the edge-free ablation model by 12%, proving the value of explicit spatial-semantic edges.
  • Temporal Boost: The Recurrent15 model (using 15 historical frames) achieved a 70% to 73% improvement over the baselines on the INTERACTION dataset, dropping the L1 acceleration error down to 0.170.
  • Dataset Characteristics: The model performed better on the INTERACTION dataset than on PandaSet. The authors attribute this to the fact that PandaSet scenes are noisier, containing roughly 24 participants per scene with 12 relations each, compared to INTERACTION's cleaner average of 5 vehicles and 2 relations per scene.
Model Dataset L1 Loss MSE Loss
Single Step INTERACTION 0.494 0.451
Single Step (no edges) INTERACTION 0.552 0.538
Recurrent15 INTERACTION 0.170 0.085
Baseline Zero INTERACTION 0.599 0.622

Limitations and Future Work

While promising, the approach currently has a few notable limitations:

  • Longitudinal Only: The model only predicts acceleration/deceleration (longitudinal control) and completely ignores steering (lateral control).
  • Missing Context: It does not yet incorporate static scene elements like road curvature, traffic lights, or stop lines directly into the graph.
  • Simplified Traffic: Vulnerable road users like pedestrians and cyclists are not yet integrated into the semantic relation framework.

The authors plan to expand the graph schema to include these missing elements and explore how these structured scene graph embeddings can be fused with state-of-the-art image-based trajectory predictors to create more robust, hybrid models.

Summary updated

Abstract

Representing relevant information of a traffic scene and understanding its environment is crucial for the success of autonomous driving. Modeling the surrounding of an autonomous car using semantic relations, i.e., how different traffic participants relate in the context of traffic rule based behaviors, is hardly been considered in previous work. This stems from the fact that these relations are hard to extract from real-world traffic scenes. In this work, we model traffic scenes in a form of spatial semantic scene graphs for various different predictions about the traffic participants, e.g., acceleration and deceleration. Our learning and inference approach uses Graph Neural Networks (GNNs) and shows that incorporating explicit information about the spatial semantic relations between traffic participants improves the predicdtion results. Specifically, the acceleration prediction of traffic participants is improved by up to 12% compared to the baselines, which do not exploit this explicit information. Furthermore, by including additional information about previous scenes, we achieve 73% improvements.
BibTeX
@misc{zipfl2022relation,
  author = {Maximilian Zipfl and Felix Hertlein and Achim Rettinger and Steffen Thoma and Lavdim Halilaj and Juergen Luettin and Stefan Schmid and Cory Henson},
  title = {{Relation-based Motion Prediction using Traffic Scene Graphs}},
  year = {2022},
  eprint = {2212.02503},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.02503},
}

Loading related content...

Figure