Abstract
Autonomous driving in complex urban environments requires planning systems that can handle multimodal future uncertainties while ensuring safety and comfort. This paper presents the Horizon Robotics team's 2nd place solution for the 2023 NuPlan Challenge. The proposed system, named hoplan, utilizes a two-stage approach: a neural network predictor that generates spatial-temporal heatmaps to capture multimodal intent, followed by a trajectory post-solver that refines the plan for kinematic feasibility and safety. Our approach achieved top scores in ego progress and passenger comfort metrics within the large-scale NuPlan benchmark.
Introduction
Autonomous driving planning is a significant challenge due to the inherent uncertainty of the future. In busy urban intersections, there is rarely a single "correct" path; instead, the future is multimodal. Traditional imitation learning, or behavior cloning, often fails in these scenarios by producing "averaged" trajectories that may be unsafe or unrealistic.
The NuPlan Challenge by Motional serves as a rigorous benchmark for these systems, providing a large-scale, closed-loop simulator. Unlike previous motion prediction tasks, NuPlan requires long-term planning (up to 15 seconds) across diverse urban scenarios. The goal is to balance "ego progress"—reaching the destination efficiently—with strict requirements for safety and passenger comfort.
Methodology
Our system architecture consists of two primary stages: a neural network-based behavior cloning module and a subsequent trajectory refinement stage using a non-linear optimizer.
1. Neural Network Predictor
The model processes a six-channel rasterized top-down image encoding the ego vehicle state, agent histories, map features, and the intended route. The architecture follows an encoder-neck-head structure:
- Encoder/Neck: A ResNet50 backbone paired with a UNet-style neck extracts multi-scale spatial features.
- Three-Head Architecture:
- Trajectory Head: Predicts a baseline initial plan (x, y, heading).
- Heatmap Head: Predicts a 3D spatial-temporal probability distribution. This heatmap allows the model to express uncertainty and represent multiple potential paths simultaneously.
- Occupancy Head: Predicts the future occupancy of other agents to facilitate collision avoidance.
2. Post-Solver Motion Planner
To ensure the final trajectory is smooth and safe, we employ the CasADi ipopt solver. The solver minimizes a multi-objective cost function that balances several factors:
- Imitation: Minimizing deviation from the network's initial trajectory.
- Guidance: Aligning the trajectory with high-probability regions in the spatial-temporal heatmap.
- Safety: Maintaining distance from predicted agent occupancy.
- Comfort: Penalizing high jerk and lateral acceleration to ensure a smooth ride.
Experimental Results
Dataset and Training
The model was trained on the NuPlan dataset, comprising approximately 1,300 hours of human driving data. We sampled 1.5 million frames across various scenario types to ensure broad coverage of driving behaviors. To improve robustness, we introduced "perturbations" during training—intentionally adding noise to the ego vehicle's starting state—to teach the model how to recover and steer back to a safe path.
Performance and Ablation Studies
Our solution achieved 2nd place overall in the NuPlan Challenge with a final score of 0.876. Notably, our system led the competition in the ego progress and comfort metrics.
Ablation studies confirmed the efficacy of our design choices. As shown in the table below, the inclusion of the spatial-temporal heatmap and the post-solver provided significant performance gains, particularly in closed-loop reactive (cl-r) and non-reactive (cl-nr) scenarios.
Specifically, adding the post-solver improved the non-reactive closed-loop score from 0.836 to 0.896, demonstrating its critical role in refining raw neural network outputs into high-quality plans.
Discussion and Conclusion
The success of hoplan highlights the value of combining the expressive power of spatial-temporal heatmaps with the rigorous constraints of a classical optimizer. By predicting heatmaps rather than single trajectories, the model effectively communicates multimodal uncertainty to the planner.
Future work will focus on reducing the inference latency associated with non-linear optimization to better suit real-time production environments. Additionally, we aim to explore more tightly integrated end-to-end architectures where the solver's constraints are differentiable and can be incorporated directly into the neural network's training loop.
Summary updated