GoRela: Go Relative for Viewpoint-Invariant Motion Forecasting

Alexander Cui, Sergio Casas, Kelvin Wong, Simon Suo, Raquel Urtasun

Abstract

The task of motion forecasting is critical for self-driving vehicles (SDVs) to plan safe maneuvers. Modern approaches reason about the map, agents' past trajectories, and their interactions to produce accurate forecasts. However, the predominant per-agent encoding approach is computationally expensive, while shared-frame encoding is often sample inefficient and vulnerable to domain shift. We propose GoRela, an efficient shared encoding for all agents and the map that leverages pair-wise relative positional encodings in a heterogeneous spatial graph. This parameterization is invariant to scene viewpoint and allows for offline map embedding computation. Our viewpoint-agnostic decoder predicts agent goals on the lane graph to enable diverse and context-aware multimodal prediction. We demonstrate the effectiveness of our approach on the Argoverse 2 benchmark and a novel highway dataset.

Introduction

Predicting the future trajectories of surrounding agents is a fundamental challenge for autonomous driving. Current methodologies often face a "viewpoint dilemma." High-accuracy models typically perform inference from the perspective of each target agent, which scales poorly as the number of agents increases. Conversely, models using a shared coordinate frame (e.g., centered on the SDV) are more efficient but struggle with generalization; unusual SDV maneuvers can create "unseen" viewpoints that degrade performance.

GoRela addresses this by introducing a framework that is mathematically invariant to the coordinate frame. By focusing on relative geometric relationships, the model maintains consistent performance regardless of the scene's global orientation or the SDV's specific pose.

Methodology

The GoRela architecture consists of three main components: relative scene encoding, goal prediction, and trajectory generation.

The GoRela pipeline: (Left) Pair-wise relative encoding of the scene. (Middle) Lane-graph relative goal prediction. (Right) Viewpoint-agnostic trajectory predictions.

Pair-wise Relative Positional Encoding (PairPose)

Instead of absolute coordinates, GoRela utilizes PairPose to represent the world through distances and angles between entities. For any two nodes (agents or map elements), the model encodes their relative distance, relative heading, and the angle of the vector between them. Using trigonometric functions of these relative values ensures the input is viewpoint-invariant.

Heterogeneous Graph Neural Network

The scene is represented as a heterogeneous graph where nodes represent agents, lane segments, or crosswalks. We employ Heterogeneous Message Passing (HMP) layers to process these relationships. This allows the model to learn distinct interaction patterns, such as how a car responds to another car versus how it follows a lane boundary.

Technical details of the Pair-wise relative heterogeneous GNN (left) and the PairPose geometric encoding parameters (right).

Efficiency and Goal-Based Decoding

Because the map-to-map relationships are static and relative, map embeddings can be precomputed offline and cached. During real-time inference, the system only updates agent-related edges, significantly reducing latency. The decoder then predicts a "goal heatmap" on the lane graph, from which goals are sampled to generate diverse, map-constrained trajectories.

Results

GoRela was evaluated on the Argoverse 2 urban dataset and a new HighwaySim dataset introduced by the authors.

  • Performance and Robustness: GoRela achieved state-of-the-art results on Argoverse 2, particularly in the BrierMinFDE metric. Crucially, while baseline models' error rates spiked when the input scene was rotated, GoRela's performance remained perfectly stable, confirming its viewpoint invariance.
  • Sample Efficiency: A key finding is GoRela's extreme sample efficiency. The model outperformed standard baselines while using only 10% of the training data, demonstrating that relative geometric reasoning simplifies the learning task.
  • Highway Navigation: In high-speed scenarios (HighwaySim), GoRela showed significant improvements in Cross Track Error (CTE), indicating a superior understanding of lane-following constraints compared to global-frame models.

Conclusion

GoRela demonstrates that viewpoint invariance is a powerful inductive bias for motion forecasting. By shifting from global coordinates to relative geometric relationships, we achieve a model that is both computationally efficient and highly robust to novel scenarios. Future work will focus on refining trajectory precision in complex maneuvers, such as tight left turns where current models may occasionally cut corners.

Summary updated

Abstract

The task of motion forecasting is critical for self-driving vehicles (SDVs) to be able to plan a safe maneuver. Towards this goal, modern approaches reason about the map, the agents' past trajectories and their interactions in order to produce accurate forecasts. The predominant approach has been to encode the map and other agents in the reference frame of each target agent. However, this approach is computationally expensive for multi-agent prediction as inference needs to be run for each agent. To tackle the scaling challenge, the solution thus far has been to encode all agents and the map in a shared coordinate frame (e.g., the SDV frame). However, this is sample inefficient and vulnerable to domain shift (e.g., when the SDV visits uncommon states). In contrast, in this paper, we propose an efficient shared encoding for all agents and the map without sacrificing accuracy or generalization. Towards this goal, we leverage pair-wise relative positional encodings to represent geometric relationships between the agents and the map elements in a heterogeneous spatial graph. This parameterization allows us to be invariant to scene viewpoint, and save online computation by re-using map embeddings computed offline. Our decoder is also viewpoint agnostic, predicting agent goals on the lane graph to enable diverse and context-aware multimodal prediction. We demonstrate the effectiveness of our approach on the urban Argoverse 2 benchmark as well as a novel highway dataset.
BibTeX
@inproceedings{cui2023gorela,
  author = {Alexander Cui and Sergio Casas and Kelvin Wong and Simon Suo and Raquel Urtasun},
  title = {{GoRela: Go Relative for Viewpoint-Invariant Motion Forecasting}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=XUS1ZUcH7uF},
}
Figure