UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving

Yuping Wang, Xiangyu Huang, Xiaokang Sun, Mingxuan Yan, Shuo Xing, Zhengzhong Tu, Jiachen Li

Abstract

We introduce UniOcc, a comprehensive, unified benchmark for occupancy forecasting (i.e., predicting future occupancies based on historical information) and current-frame occupancy prediction from camera images. UniOcc unifies data from multiple real-world datasets (i.e., nuScenes, Waymo) and high-fidelity driving simulators (i.e., CARLA, OpenCOOD), which provides 2D/3D occupancy labels with per-voxel flow annotations and support for cooperative autonomous driving. In terms of evaluation, unlike existing studies that rely on suboptimal pseudo labels for evaluation, UniOcc incorporates novel metrics that do not depend on ground-truth occupancy, enabling robust assessment of additional aspects of occupancy quality. Through extensive experiments on state-of-the-art models, we demonstrate that large-scale, diverse training data and explicit flow information significantly enhance occupancy prediction and forecasting performance.

Introduction

In autonomous driving, Occupancy Grid Maps (OGMs) have emerged as a powerful representation, discretizing 3D space into voxels that are either occupied or free. Despite their popularity, the field has been hindered by fragmented data formats across platforms like nuScenes and Waymo, suboptimal "pseudo-labels" generated from sparse LiDAR that fail to capture complete object geometry, and a lack of fine-grained motion information. UniOcc addresses these challenges by providing the first unified benchmark and toolkit designed to standardize 3D occupancy forecasting and prediction across both real-world and simulated environments.

The UniOcc Framework

The UniOcc framework unifies data from four major sources—nuScenes, Waymo, CARLA, and OpenCOOD—into a single, streamlined pipeline. This unification allows for large-scale training that generalizes across different cities and sensor configurations.

The UniOcc Framework: A unified pipeline from data generation to evaluation.

Voxel-Level Motion Flow

A core technical contribution of UniOcc is the introduction of voxel-level motion flow. While previous methods often assigned a single velocity vector to an entire object, UniOcc annotates unique 3D displacement for every individual voxel. This granularity is essential for capturing complex dynamics, such as the differential movement of voxels during a vehicle's rotation or turn.

Voxel-Level Flow vs. Object-Level Flow: Voxel-level flow captures complex rotations that object-level flow misses.

Unified Evaluation Metrics

To overcome the limitations of flawed ground-truth labels, UniOcc introduces "ground-truth-free" metrics:

  • Key Object Dimension Probability ($P_{car}$): Uses Gaussian Mixture Models (GMMs) to assess if predicted objects have realistic physical dimensions.
  • Temporal Consistency: Measures the stability of an object's shape across consecutive frames, penalizing predictions that "flicker" or morph unnaturally.

Experimental Insights

Extensive testing on state-of-the-art models yielded several key insights:

  1. Flow is Essential for Forecasting: Explicitly modeling voxel-level flow consistently improves performance. For instance, adding flow to the OccWorld model on nuScenes improved the 3-second mIoU from 18.13 to 19.06.
  2. The Power of Diversity: Models trained on a unified mix of datasets performed significantly better on Out-of-Distribution (OOD) tests. A model trained only on nuScenes saw its IoU drop from 62.62 to 49.32 when tested on Waymo, whereas the unified model maintained high performance across both.
  3. Simulation as a "Shape Teacher": While real-world labels are often incomplete due to LiDAR occlusion, simulation data from CARLA provides perfect geometry. Incorporating this data helps models learn to predict fuller, more realistic object shapes.
  4. Cooperative Gains: In multi-agent scenarios (OpenCOOD), vehicle-to-vehicle cooperation significantly boosts occupancy prediction, reaching an IoU of 87.22 for car instances by allowing vehicles to "see" around corners.

Conclusion

UniOcc standardizes the occupancy task, proving that large-scale diverse data and explicit motion modeling are the keys to robust autonomous perception. While challenges remain—such as calibrating the speed mismatch between simulated and real-world agents—UniOcc provides the necessary infrastructure for the next generation of predictive driving systems. Future work will explore integrating Large Vision-Language Models (VLMs) to provide semantic feedback and further refine temporal consistency.

Summary updated

Abstract

We introduce UniOcc, a comprehensive, unified benchmark for occupancy forecasting (i.e., predicting future occupancies based on historical information) and current-frame occupancy prediction from camera images. UniOcc unifies data from multiple real-world datasets (i.e., nuScenes, Waymo) and high-fidelity driving simulators (i.e., CARLA, OpenCOOD), which provides 2D/3D occupancy labels with per-voxel flow annotations and support for cooperative autonomous driving. In terms of evaluation, unlike existing studies that rely on suboptimal pseudo labels for evaluation, UniOcc incorporates novel metrics that do not depend on ground-truth occupancy, enabling robust assessment of additional aspects of occupancy quality. Through extensive experiments on state-of-the-art models, we demonstrate that large-scale, diverse training data and explicit flow information significantly enhance occupancy prediction and forecasting performance.
BibTeX
@inproceedings{wang2025uniocc,
  author = {Yuping Wang and Xiangyu Huang and Xiaokang Sun and Mingxuan Yan and Shuo Xing and Zhengzhong Tu and Jiachen Li},
  title = {{UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure