LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting

Wenyuan Zeng, Ming Liang, Renjie Liao, Raquel Urtasun

Predicting where a car will go in the next five seconds is a bit like playing chess on a board that’s constantly changing. Traditional autonomous driving models try to solve this by turning everything—the car’s history, the surrounding traffic, and the map—into a single, dense vector of numbers. But as any driver knows, your future isn't just a function of your speed; it’s dictated by the "plumbing" of the road: the lanes, the intersections, and the traffic lights.

In their paper LaneRCNN, researchers from the University of Toronto and Uber ATG argue that we should stop treating actors as isolated points and start treating them as part of the map's graph.

TL;DR

LaneRCNN introduces a graph-centric approach to motion forecasting by representing each actor as a "LaneRoI"—a local subgraph of the map. By modeling interactions through a shared global lane graph and using lanes as anchors for prediction, the model achieved 1st place on the Argoverse leaderboard with superior data efficiency.

What Problem Does This Paper Address?

Most state-of-the-art motion forecasting models share three major flaws:

  1. Information Bottlenecks: Trying to squeeze 100 meters of road context and five seconds of history into a single feature vector is incredibly difficult.
  2. Blind Interactions: Many models use "fully-connected" interaction graphs, meaning every car "talks" to every other car. In reality, a car on the opposite side of a concrete divider doesn't influence your path nearly as much as the one merging into your lane.
  3. Unrealistic Predictions: Standard regression headers often predict trajectories that "shoot out of road" because they don't explicitly use the lane geometry as a guide.

Methods

The core innovation is the Lane-graph Region-of-Interest (LaneRoI). Instead of a feature vector, each actor is represented by a subgraph of the lanes they are likely to occupy.

  • LaneRoI Representation: For every actor, the model retrieves relevant lane segments. Each node in this local graph encodes the actor's past motion relative to that specific lane segment, along with semantic data (like "is this a turn lane?").
  • Graph-Centric Interaction: To model how cars affect each other, the model embeds these LaneRoIs into a shared global lane graph. Information flows along the map's topology. If two actors share a lane or have connecting lanes, they communicate more frequently during the message-passing phase.
  • Map-Relative Decoding: Instead of predicting raw $(x, y)$ coordinates, the model treats lane segments as "anchors." It first classifies which lane segment is the most likely "goal" and then regresses the precise position relative to that segment.
  • Bezier Refinement: The model fits a Bezier quadratic curve to the predicted goal to ensure a smooth, kinematically feasible path, which is then refined by a final learnable header.

Datasets

The authors evaluated LaneRCNN on the Argoverse Motion Forecasting Benchmark.

  • Scale: Over 30,000 real-world driving sequences from Miami and Pittsburgh.
  • Task: Predict 3 seconds of future motion based on 2 seconds of history.
  • Context: Includes rich HD map information (lane centerlines, connectivity, etc.).

Key Insights and Ideas

  • Distributed is Better than Unitary: By spreading an actor's representation across a graph (LaneRoI) rather than a single vector, the model preserves fine-grained spatial relationships that are usually lost.
  • The Map is the Message: Interactions are most meaningful when they follow the road's structure. By performing message passing over the lane graph, the model naturally ignores irrelevant actors and focuses on those that share a path.
  • Lanes as Inductive Bias: Using lanes as anchors makes the learning process much easier. The model doesn't have to "learn" that cars stay in lanes; that knowledge is baked into the output space. This makes the model significantly more data-efficient.

Key Results

  • Leaderboard Dominance: At the time of publication, LaneRCNN held 1st place on the Argoverse leaderboard, significantly outperforming previous winners like TNT and Jean.
  • Miss Rate (MR): Achieved a Miss Rate of 12.3% (at K=6), a notable improvement over the previous best of 13.1%.
  • Data Efficiency: Perhaps the most striking result is that LaneRCNN with only 1% of the training data achieved a lower Miss Rate (~20%) than a standard regression-based model using 100% of the data.
  • Qualitative Success: The model successfully predicts complex maneuvers like lane changes, overtaking, and navigating sharp intersections where previous models often failed.

Limitations and Future Work

  • Heuristic Dependency: The construction of the LaneRoI currently relies on a simple distance-based heuristic to pick "relevant" lanes. A more learned approach to lane association could improve robustness.
  • Map Accuracy: Like most map-heavy models, LaneRCNN is sensitive to errors in the HD map. While the authors show some qualitative examples of the model handling "off-map" behavior, a formal study on its resilience to map noise would be valuable.
  • Computational Complexity: Processing a graph for every actor is more computationally expensive than simple vector operations, which might pose challenges for real-time deployment on edge hardware.

Summary updated

Abstract

Forecasting the future behaviors of dynamic actors is an important task in many robotics applications such as self-driving. It is extremely challenging as actors have latent intentions and their trajectories are governed by complex interactions between the other actors, themselves, and the map. In this paper, we propose LaneRCNN, a graph-centric motion forecasting model that captures the actor-to-actor and the actor-to-map relations in a distributed and structured manner. Relying on a specially designed graph encoder, we learn a local graph representation per actor (LaneRoI) to encode its past motions and the local map topology. We further develop an interaction module which permits efficient message passing among local graph representations within a shared global lane graph. Moreover, we parameterize the output trajectories based on lane graphs, a more amenable prediction parameterization. We demonstrate the effectiveness of our approach on the challenging Argoverse [1] motion forecasting benchmark and achieve state-of-the-art performance.
BibTeX
@inproceedings{zeng2021lanercnn,
  author = {Wenyuan Zeng and Ming Liang and Renjie Liao and Raquel Urtasun},
  title = {{LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=nCLW89e0AZ},
}

Loading related content...

Figure