Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data

Sergio Casas, Cole Gulino, Renjie Liao, Raquel Urtasun

Abstract

In this paper, we tackle the problem of relational behavior forecasting from sensor data. Towards this goal, we propose a novel spatially-aware graph neural network (SpAGNN) that models the interactions between agents in the scene. Specifically, we exploit a convolutional neural network to detect the actors and compute their initial states. A graph neural network then iteratively updates the actor states via a message passing process. Inspired by Gaussian belief propagation, we design the messages to be spatially-transformed parameters of the output distributions from neighboring agents. Our model is fully differentiable, thus enabling end-to-end training. Importantly, our probabilistic predictions can model uncertainty at the trajectory level. We demonstrate the effectiveness of our approach by achieving significant improvements over the state-of-the-art on two real-world self-driving datasets: ATG4D and nuScenes.

Introduction

Predicting the future behavior of human drivers is a fundamental challenge for autonomous driving. Current systems frequently encounter two significant hurdles. First, the "pipeline gap" occurs when detection and prediction are treated as separate, sequential modules; errors in detection can propagate and cause failures in forecasting. Second, "social blindness" arises when models predict trajectories for individual agents in a vacuum, failing to account for the complex negotiations and spatial interactions that occur in real-world traffic.

To address these challenges, we propose SpAGNN (Spatially-Aware Graph Neural Networks), a framework that jointly handles object detection and motion forecasting. By treating detected vehicles as nodes in a graph and passing "spatially-aware" messages between them, the model explicitly accounts for social interactions, leading to significantly more accurate and safer trajectory predictions.

Methodology

The SpAGNN architecture consists of two primary stages integrated into a single, end-to-end differentiable model.

Multi-Stream Detection

The model utilizes a two-stream Convolutional Neural Network (CNN) backbone. One stream processes voxelized LiDAR point clouds in a Bird’s Eye View (BEV), while the other processes high-definition (HD) maps. These features are fused to detect actors and provide an initial estimate of their states.

Spatially-Aware Graph Neural Networks

Once actors are detected, they are represented as nodes in a fully connected graph. The model then performs iterative message passing to refine these states.

The SpAGNN architecture: A multi-stream CNN detects actors, followed by RRoI Align and iterative message passing within the SpAGNN block to refine trajectory distributions.

A key innovation is the Spatial Awareness of the messages. Standard GNNs often struggle with global spatial relationships. SpAGNN overcomes this by transforming messages into the local coordinate system of the receiving vehicle. This allows the network to learn relative geometric concepts (e.g., "the vehicle behind is accelerating") invariant of global position. Furthermore, inspired by Gaussian Belief Propagation, the messages are parameters of probability distributions, allowing the model to iteratively refine its uncertainty estimates.

Experimental Results

The proposed model was evaluated on two large-scale datasets: ATG4D (high-resolution 64-beam LiDAR) and nuScenes (32-beam LiDAR).

Quantitative Performance

SpAGNN outperformed state-of-the-art models across nearly all metrics. On the ATG4D dataset, it achieved a 15% improvement in L2 displacement accuracy at the 3-second mark compared to joint models like IntentNet.

Precision-Recall and Collision Rate curves: SpAGNN demonstrates superior detection performance and a significantly lower collision rate compared to baselines.

As shown in the performance curves, SpAGNN maintains high precision and recall while drastically reducing the predicted collision rate. Specifically, the collision rate at the 3-second horizon was reduced from approximately 3.06% in the best baseline to just 0.42%.

Social Coherence

By modeling interactions, the system produces "socially coherent" trajectories. In dense traffic, SpAGNN is significantly less likely to predict overlapping trajectories (collisions) compared to non-relational baselines, as it learns to respect the physical space occupied by other agents.

Conclusion

SpAGNN demonstrates that joint optimization of detection and prediction, combined with spatially-aware relational modeling, is essential for robust behavior forecasting. While the current model focuses on single-mode behavior for vehicles, future work will explore multi-modal predictions and the inclusion of diverse actor types such as pedestrians and cyclists.

Summary updated

Abstract

In this paper, we tackle the problem of relational behavior forecasting from sensor data. Towards this goal, we propose a novel spatially-aware graph neural network (SpAGNN) that models the interactions between agents in the scene. Specifically, we exploit a convolutional neural network to detect the actors and compute their initial states. A graph neural network then iteratively updates the actor states via a message passing process. Inspired by Gaussian belief propagation, we design the messages to be spatially-transformed parameters of the output distributions from neighboring agents. Our model is fully differentiable, thus enabling end-to-end training. Importantly, our probabilistic predictions can model uncertainty at the trajectory level. We demonstrate the effectiveness of our approach by achieving significant improvements over the state-of-the-art on two real-world self-driving datasets: ATG4D and nuScenes.
BibTeX
@inproceedings{casas2020spatially,
  author = {Sergio Casas and Cole Gulino and Renjie Liao and Raquel Urtasun},
  title = {{Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=16Bw_RH44Vy},
}
Figure