Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challenge

Yihan Hu, Kun Li, Pingyuan Liang, Jingyu Qian, Zhening Yang, Haichao Zhang, Wenxin Shao, Zhuangzhuang Ding, Wei Xu, Qiang LiuarXiv 2023 · DOI · arXiv

Abstract

Autonomous driving in complex urban environments requires planning systems that can handle multimodal future uncertainties while ensuring safety and comfort. This paper presents the Horizon Robotics team's 2nd place solution for the 2023 NuPlan Challenge. The proposed system, named hoplan, utilizes a two-stage approach: a neural network predictor that generates spatial-temporal heatmaps to capture multimodal intent, followed by a trajectory post-solver that refines the plan for kinematic feasibility and safety. Our approach achieved top scores in ego progress and passenger comfort metrics within the large-scale NuPlan benchmark.

Introduction

Autonomous driving planning is a significant challenge due to the inherent uncertainty of the future. In busy urban intersections, there is rarely a single "correct" path; instead, the future is multimodal. Traditional imitation learning, or behavior cloning, often fails in these scenarios by producing "averaged" trajectories that may be unsafe or unrealistic.

The NuPlan Challenge by Motional serves as a rigorous benchmark for these systems, providing a large-scale, closed-loop simulator. Unlike previous motion prediction tasks, NuPlan requires long-term planning (up to 15 seconds) across diverse urban scenarios. The goal is to balance "ego progress"—reaching the destination efficiently—with strict requirements for safety and passenger comfort.

Methodology

Our system architecture consists of two primary stages: a neural network-based behavior cloning module and a subsequent trajectory refinement stage using a non-linear optimizer.

1. Neural Network Predictor

The model processes a six-channel rasterized top-down image encoding the ego vehicle state, agent histories, map features, and the intended route. The architecture follows an encoder-neck-head structure:

  • Encoder/Neck: A ResNet50 backbone paired with a UNet-style neck extracts multi-scale spatial features.
  • Three-Head Architecture:
    • Trajectory Head: Predicts a baseline initial plan (x, y, heading).
  • Heatmap Head: Predicts a 3D spatial-temporal probability distribution. This heatmap allows the model to express uncertainty and represent multiple potential paths simultaneously.
  • Occupancy Head: Predicts the future occupancy of other agents to facilitate collision avoidance.
System architecture of the proposed two-stage motion planner.

2. Post-Solver Motion Planner

To ensure the final trajectory is smooth and safe, we employ the CasADi ipopt solver. The solver minimizes a multi-objective cost function that balances several factors:

  • Imitation: Minimizing deviation from the network's initial trajectory.
  • Guidance: Aligning the trajectory with high-probability regions in the spatial-temporal heatmap.
  • Safety: Maintaining distance from predicted agent occupancy.
  • Comfort: Penalizing high jerk and lateral acceleration to ensure a smooth ride.

Experimental Results

Dataset and Training

The model was trained on the NuPlan dataset, comprising approximately 1,300 hours of human driving data. We sampled 1.5 million frames across various scenario types to ensure broad coverage of driving behaviors. To improve robustness, we introduced "perturbations" during training—intentionally adding noise to the ego vehicle's starting state—to teach the model how to recover and steer back to a safe path.

Performance and Ablation Studies

Our solution achieved 2nd place overall in the NuPlan Challenge with a final score of 0.876. Notably, our system led the competition in the ego progress and comfort metrics.

Ablation studies confirmed the efficacy of our design choices. As shown in the table below, the inclusion of the spatial-temporal heatmap and the post-solver provided significant performance gains, particularly in closed-loop reactive (cl-r) and non-reactive (cl-nr) scenarios.

Ablation study results showing the impact of perturbations, heatmaps, and the post-solver.

Specifically, adding the post-solver improved the non-reactive closed-loop score from 0.836 to 0.896, demonstrating its critical role in refining raw neural network outputs into high-quality plans.

Discussion and Conclusion

The success of hoplan highlights the value of combining the expressive power of spatial-temporal heatmaps with the rigorous constraints of a classical optimizer. By predicting heatmaps rather than single trajectories, the model effectively communicates multimodal uncertainty to the planner.

Future work will focus on reducing the inference latency associated with non-linear optimization to better suit real-time production environments. Additionally, we aim to explore more tightly integrated end-to-end architectures where the solver's constraints are differentiable and can be incorporated directly into the neural network's training loop.

Summary updated

Abstract

This paper presents our 2nd place solution for the NuPlan Challenge 2023. Autonomous driving in real-world scenarios is highly complex and uncertain. Achieving safe planning in the complex multimodal scenarios is a highly challenging task. Our approach, Imitation with Spatial-Temporal Heatmap, adopts the learning form of behavior cloning, innovatively predicts the future multimodal states with a heatmap representation, and uses trajectory refinement techniques to ensure final safety. The experiment shows that our method effectively balances the vehicle's progress and safety, generating safe and comfortable trajectories. In the NuPlan competition, we achieved the second highest overall score, while obtained the best scores in the ego progress and comfort metrics.
BibTeX
@misc{hu2023imitation,
  author = {Yihan Hu and Yihan Hu and Kun Li and Pingyuan Liang and Jingyu Qian and Zhening Yang and Haichao Zhang and Wenxin Shao and Zhuangzhuang Ding and Wei Xu and Qiang Liu},
  title = {{Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challenge}},
  year = {2023},
  eprint = {2306.15700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.15700},
}
Figure