Self-Supervised Simultaneous Multi-Step Prediction of Road Dynamics and Cost Map

Elmira Amirloo Abolfathi, Mohsen Rohani, Ershad Banijamali, Jun Luo, Pascal Poupart

Abstract

In this paper we propose a system consisting of a modular network and a trajectory planner. The network simultaneously predicts Occupancy Grid Maps (OGMs) and estimates space-time cost maps (CMs) corresponding to the areas around the vehicle. The trajectory planner computes the cost of a set of predefined trajectories and chooses the one with the lowest cost. Training this network is done in a self-supervised manner which desirably do not require any labeled data. The proposed training objective takes into account the accuracy of OGM predictions as well as contextual information and human driver behavior. Training these modules end-to-end makes each module aware of the errors caused by the other components of the system. We show that our proposed method can lead to the selection of low cost trajectories with a low collision rate and road violation in fairly long planning horizons.

Introduction

Autonomous driving (AD) systems typically follow one of two paradigms: modular stacks or end-to-end systems. Modular stacks offer interpretability through separate perception, prediction, and planning components but rely heavily on expensive human-labeled data and complex, hand-tuned cost functions. Conversely, end-to-end systems are highly scalable as they learn directly from raw data, but their "black box" nature lacks the transparency required for safety-critical applications.

This research addresses the interpretability-scalability trade-off by proposing a self-supervised framework. The system generates interpretable intermediate representations—Occupancy Grid Maps (OGMs) and Cost Maps (CMs)—while learning from raw sensor data without manual labels. This approach combines the transparency of modular stacks with the scalability of end-to-end learning.

Methods

The core of the framework consists of two neural architectures: the Recurrent Cost Map Estimator (RCME) and the Multi-Step Cost Map Estimator (MSCME). The model processes a sequence of LiDAR point clouds and HD map information to simultaneously predict future OGMs (a bird's-eye view of obstacle dynamics) and estimate space-time cost maps.

High-dimensional cost estimation framework with auxiliary imitation tasks

Self-Supervised Learning Objectives

Since ground-truth cost maps are unavailable, the system is trained using two primary signals:

  1. Environmental Priors: A loss term that penalizes the model for assigning low costs to occupied cells or non-drivable areas defined by the HD map.
  2. Auxiliary Imitation Task: This key innovation forces the model to predict the intentions and trajectories of an expert human driver. By ensuring that a simple planner using the estimated cost maps can recreate the human's path, the model learns the underlying cost structure of the environment.

Results

The framework was evaluated using the Argoverse dataset. The researchers converted raw data into $256 \times 256$ bird's-eye view (BEV) grids. The results indicate that simultaneous learning of OGMs and Cost Maps is superior to learning either in isolation, as OGM predictions provide mid-level features that aid in reasoning about moving objects.

Performance comparison of the proposed CME models against baselines across different planning horizons

As shown in the quantitative evaluation, the proposed CME models significantly outperform baselines like Behavior Cloning (BC) in safety metrics. While BC may achieve lower displacement errors (minADE) by mimicking exact paths, it suffers from much higher Potential Collision Rates (CR) and Road Violations (RV) in complex scenarios. The CME approach prioritizes safety and legality, maintaining low violation rates even in long-range planning horizons (e.g., $T=3$ seconds).

Conclusion

This work demonstrates a successful self-supervised approach to autonomous planning that maintains interpretability through cost maps and occupancy grids. By learning the "why" behind driving decisions rather than just copying steering commands, the system achieves greater robustness. Future research will focus on modeling multi-agent interactions, where the ego-vehicle's actions influence the behavior of others, and reducing the reliance on high-definition maps.

Summary updated

Abstract

In this paper we propose a system consisting of a modular network and a trajectory planner. The network simultaneously predicts Occupancy Grid Maps (OGMs) and estimates space-time cost maps (CMs) corresponding to the areas around the vehicle. The trajectory planner computes the cost of a set of predefined trajectories and chooses the one with the lowest cost. Training this network is done in a self-supervised manner which desirably do not require any labeled data. The proposed training objective takes into account the accuracy of OGM predictions as well as contextual information and human driver behavior. Training these modules end-to-end makes each module aware of the errors caused by the other components of the system. We show that our proposed method can lead to the selection of low cost trajectories with a low collision rate and road violation in fairly long planning horizons.
BibTeX
@inproceedings{abolfathi2021self,
  author = {Elmira Amirloo Abolfathi and Mohsen Rohani and Ershad Banijamali and Jun Luo and Pascal Poupart},
  title = {{Self-Supervised Simultaneous Multi-Step Prediction of Road Dynamics and Cost Map}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=vo84M12m37},
}
Figure