Safe Local Motion Planning With Self-Supervised Freespace Forecasting

Peiyun Hu, Aaron Huang, John M. Dolan, David Held, Deva RamananCVPR 2021 · OpenReview

Abstract

Safe local motion planning for autonomous driving in dynamic environments requires forecasting how the scene evolves. Practical autonomy stacks adopt a semantic object-centric representation of a dynamic scene and build object detection, tracking, and prediction modules to solve forecasting. However, training these modules comes at an enormous human cost of manually annotated objects across frames. In this work, we explore future freespace as an alternative representation to support motion planning. Our key intuition is that it is important to avoid straying into occupied space regardless of what is occupying it. Importantly, computing ground-truth future freespace is annotation-free. First, we explore freespace forecasting as a self-supervised learning task. We then demonstrate how to use forecasted freespace to identify collision-prone plans from off-the-shelf motion planners. Finally, we propose future freespace as an additional source of annotation-free supervision. We demonstrate how to integrate such supervision into the learning-based planners. Experimental results on nuScenes and CARLA suggest both approaches lead to a significant reduction in collision rates.

Introduction

Standard autonomous vehicle (AV) perception stacks rely on detecting, tracking, and predicting the trajectories of specific objects such as cars, cyclists, and pedestrians. While effective, this object-centric approach faces a significant bottleneck: the high cost of manual annotation. Training these models requires humans to label millions of frames with 3D bounding boxes. Furthermore, these models often struggle with "long-tail" scenarios involving unusual objects or behaviors not well-represented in training data.

This research proposes a shift from expensive, human-labeled object-centric models to self-supervised "freespace-centric" forecasting. The core argument is that for safe local motion planning, the most critical information is not the identity of an obstacle, but simply its location. By forecasting how "freespace" (empty areas) evolves over time, we can create a representation that is naturally annotation-free, as ground truth can be automatically generated from raw LiDAR data as time progresses.

Methodology

The proposed framework for Freespace Forecasting (FF) integrates into existing motion planners through several key steps.

Freespace Computation and Representation

The system uses raycasting on LiDAR point clouds to generate 2D bird’s-eye-view (BEV) occupancy grids. Voxels are labeled as free, occupied, or unknown (occluded). This simplifies the scene into a unified representation where the goal is to avoid any non-free space, regardless of the object type.

Comparison between object-centric (left) and freespace-centric (right) scene representations.

Residual Forecasting

Rather than training a neural network to predict the entire future scene from scratch, the authors employ a "residual" approach. The model performs a simple linear extrapolation of the current scene and uses a convolutional neural network (CNN) to predict non-linear "residuals" or changes. This hybrid approach leverages geometric priors while allowing the network to learn complex dynamics, making the learning task more accurate and easier to train.

Planning Integration

The forecasted freespace is integrated into planning in two ways:

  1. Post-hoc Vetting: An off-the-shelf planner's output is "vetoed" if it leads to a predicted collision, triggering an emergency brake.
  2. Learning-based Supervision: Ground-truth future freespace, extracted from logs post-hoc, serves as an additional "margin" loss to train Inverse Optimal Control (IOC) planners, teaching them to avoid areas that will eventually be occupied.

Results and Discussion

The framework was evaluated using the CARLA simulator (NoCrash benchmark) for on-policy testing and the nuScenes dataset for real-world performance.

Performance Metrics

On the CARLA benchmark, adding freespace forecasting (LBC+FF) significantly improved success rates in dense traffic by converting potential collisions into safer "timeouts." In real-world scenarios using the nuScenes dataset, incorporating freespace supervision reduced the collision rate at a 3-second horizon from 2.76% to 1.65%. This performance nearly matches models trained with expensive human-annotated object boxes, demonstrating the power of self-supervision.

Comparison of ground-truth future freespace (a) and predicted future freespace (b) over a temporal horizon.

Key Insights

  • Self-Supervision as a "Free Lunch": Future freespace is a "free" label derived from raw LiDAR data, allowing models to train on vast amounts of data without manual labeling.
  • Universality: A freespace-centric model treats all occupied space equally, ensuring safety even when encountering unusual objects that might be missed by specific object detectors.
  • Residual Learning: The combination of geometric extrapolation and neural prediction outperformed purely geometric or purely neural variants in Average Precision (AP) and F1 scores.

Conclusion

Freespace forecasting offers a scalable and effective alternative to object-centric forecasting for AV motion planning. While the current system uses simple fallbacks like emergency braking and can occasionally produce "phantom" obstacles in quiet environments, it significantly reduces collision rates. Future work could explore more sophisticated evasive maneuvers and better integration with long-term route planning to improve recovery from near-collision states.

Summary updated

Abstract

Safe local motion planning for autonomous driving in dynamic environments requires forecasting how the scene evolves. Practical autonomy stacks adopt a semantic object-centric representation of a dynamic scene and build object detection, tracking, and prediction modules to solve forecasting. However, training these modules comes at an enormous human cost of manually annotated objects across frames. In this work, we explore future freespace as an alternative representation to support motion planning. Our key intuition is that it is important to avoid straying into occupied space regardless of what is occupying it. Importantly, computing ground-truth future freespace is annotation-free. First, we explore freespace forecasting as a self-supervised learning task. We then demonstrate how to use forecasted freespace to identify collision-prone plans from off-the-shelf motion planners. Finally, we propose future freespace as an additional source of annotation-free supervision. We demonstrate how to integrate such supervision into the learning-based planners. Experimental results on nuScenes and CARLA suggest both approaches lead to a significant reduction in collision rates.
BibTeX
@inproceedings{hu2021safe,
  author = {Peiyun Hu and Aaron Huang and John M. Dolan and David Held and Deva Ramanan},
  title = {{Safe Local Motion Planning With Self-Supervised Freespace Forecasting}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=Rmw9SQcYxiI},
}

Loading related content...

Figure