LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting

Wenyuan Zeng, Ming Liang, Renjie Liao, Raquel Urtasun

Abstract

Forecasting the future behaviors of dynamic actors is an important task in many robotics applications such as self-driving. It is extremely challenging as actors have latent intentions and their trajectories are governed by complex interactions between the other actors, themselves, and the maps. In this paper, we propose LaneRCNN, a graph-centric motion forecasting model. Importantly, relying on a specially designed graph encoder, we learn a local lane graph representation per actor (LaneRoI) to encode its past motions and the local map topology. We further develop an interaction module which permits efficient message passing among local graph representations within a shared global lane graph. Moreover, we parameterize the output trajectories based on lane graphs, a more amenable prediction parameterization. Our LaneRCNN captures the actor-to-actor and the actor-to-map relations in a distributed and map-aware manner. We demonstrate the effectiveness of our approach on the large-scale Argoverse Motion Forecasting Benchmark. We achieve the 1st place on the leaderboard and significantly outperform previous best results.

Introduction

Predicting the future trajectories of dynamic actors—such as vehicles and pedestrians—is a cornerstone of safe autonomous driving. Traditional approaches often treat actors as isolated entities, encoding their history and surroundings into a single, dense feature vector. However, this "unitary" representation creates significant information bottlenecks and often fails to capture the intricate relationship between an actor's motion and the underlying road geometry.

In contrast to the traditional paradigm where interactions are modeled as node-to-node attention between actor features, LaneRCNN proposes a graph-centric approach. As shown in Figure 1, LaneRCNN represents each actor as a local graph (LaneRoI) and models interactions as graph-to-graph communication over the map's topology.

Comparison between the traditional paradigm and LaneRCNN's graph-centric approach.

Methodology

The core of LaneRCNN is the Lane-graph Region-of-Interest (LaneRoI). Instead of a single vector, each actor is represented by a subgraph of the lanes they are likely to occupy.

1. LaneRoI Representation

For every actor, the model retrieves relevant lane segments from the HD map. Each node in this local graph encodes the actor's past motion relative to that specific lane segment, combined with semantic data (e.g., lane type). This distributed representation preserves fine-grained spatial relationships that are typically lost in global pooling.

2. Graph-Centric Interaction

To model actor-to-actor and actor-to-map interactions, LaneRCNN embeds these LaneRoIs into a shared global lane graph. Information flows along the map's topology via message passing. This ensures that actors communicate primarily with others that share or intersect their potential paths, naturally filtering out irrelevant context.

3. Map-Relative Decoding

The model avoids predicting raw $(x, y)$ coordinates directly. Instead, it treats lane segments as "anchors." It first performs per-node classification to identify the most likely "goal" lane segments and then regresses the precise position relative to those segments. A Bezier quadratic curve is fitted to the predicted goals to ensure kinematically feasible, smooth trajectories.

The LaneRCNN pipeline: from actor past trajectories and maps to multi-modal goal-based predictions.

Evaluation

The authors evaluated LaneRCNN on the Argoverse Motion Forecasting Benchmark, which contains over 30,000 real-world driving sequences.

Key Results

  • Leaderboard Performance: LaneRCNN achieved 1st place on the Argoverse leaderboard at the time of publication, outperforming previous state-of-the-art models like TNT.
  • Accuracy: It reached a Miss Rate (MR) of 12.3% at K=6, a significant improvement over the previous best of 13.1%.
  • Data Efficiency: A standout finding was the model's efficiency; LaneRCNN trained on just 1% of the data outperformed standard regression models trained on 100% of the data. This suggests that the lane-graph inductive bias significantly simplifies the learning task.

Discussion and Limitations

LaneRCNN demonstrates that a "map-first" mindset, where actors are intrinsically linked to the road graph, leads to more robust and data-efficient forecasting. However, the model's reliance on distance-based heuristics for LaneRoI construction and its sensitivity to HD map accuracy remain areas for future improvement. Additionally, the computational cost of graph-based operations per actor presents a challenge for real-time deployment on constrained hardware.

Summary updated

Abstract

Forecasting the future behaviors of dynamic actors is an important task in many robotics applications such as self-driving. It is extremely challenging as actors have latent intentions and their trajectories are governed by complex interactions between the other actors, themselves, and the maps. In this paper, we propose LaneRCNN, a graph-centric motion forecasting model. Importantly, relying on a specially designed graph encoder, we learn a local lane graph representation per actor (LaneRoI) to encode its past motions and the local map topology. We further develop an interaction module which permits efficient message passing among local graph representations within a shared global lane graph. Moreover, we parameterize the output trajectories based on lane graphs, a more amenable prediction parameterization. Our LaneRCNN captures the actor-to-actor and the actor-to-map relations in a distributed and map-aware manner. We demonstrate the effectiveness of our approach on the large-scale Argoverse Motion Forecasting Benchmark. We achieve the 1st place on the leaderboard and significantly outperform previous best results.
BibTeX
@inproceedings{zeng2021lanercnn,
  author = {Wenyuan Zeng and Ming Liang and Renjie Liao and Raquel Urtasun},
  title = {{LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=nCLW89e0AZ},
}
Figure