TL;DR
This paper introduces UniOcc, a comprehensive benchmark and toolkit designed to unify and standardize the evaluation of occupancy forecasting and prediction for autonomous driving. By integrating diverse real-world and simulated datasets, UniOcc addresses critical issues like suboptimal labels and fragmented data, enabling more robust and generalizable model development.
What Problem Does This Paper Address?
Autonomous driving systems heavily rely on understanding the surrounding environment, and occupancy grid maps (OGMs) are a popular way to represent this understanding. However, two key tasks, occupancy forecasting (predicting future occupancy) and occupancy prediction (estimating current occupancy from sensor data), face significant challenges. Existing research often suffers from:
- Suboptimal Labels: Public datasets lack direct 3D occupancy labels, forcing researchers to rely on "pseudo-labels" derived from heuristics or manual annotation. These labels often only capture reflective surfaces (like car sides) and fail to represent true 3D occupancy, leading to models that learn flawed representations.
- Fragmented Data & Domain Mismatch: Models trained on one dataset (e.g., nuScenes) often perform poorly on others (e.g., Waymo) due to differences in sensor configurations, data formats, and annotation styles. This makes cross-dataset training and generalization difficult.
- Lack of Motion Information: Standard occupancy labels typically don't include motion flow, which is crucial for forecasting dynamic scenes and understanding object movement over time.
- Limited Evaluation: Existing metrics, like Intersection-over-Union (IoU), can be misleading when applied to imperfect pseudo-labels, failing to accurately assess model quality.
UniOcc aims to solve these problems by providing a unified framework, dataset, and evaluation metrics.
Methods
UniOcc introduces a unified framework that standardizes data formats and processing pipelines for various occupancy tasks. Key methodological contributions include:
- Unified Data Format: A consistent representation for 3D voxel grids, camera images, FOV masks, camera intrinsics/extrinsics, and ego-to-world transformations.
- Per-Voxel Flow Annotations: UniOcc provides both forward and backward occupancy flow for each voxel. This is a significant advancement, as it captures full 3D motion, including rotations, which is crucial for forecasting and goes beyond previous methods that assigned single velocities to entire objects.
- Ground-Truth-Free Evaluation Metrics: To address the issue of imperfect pseudo-labels, UniOcc proposes novel metrics that assess geometric plausibility and temporal consistency without relying on ground truth. These include:
- Key Object Dimension Probability: Evaluates if predicted object dimensions are realistic for their category using Gaussian Mixture Models.
- Temporal Foreground Object Shape Consistency: Measures how stable the shape of predicted dynamic objects is across frames.
- Temporal Background Environment Consistency: Assesses the stability of static background regions over time.
- Unified Processing Toolkit: Includes tools for object segmentation and tracking directly within the voxel space, enabling more advanced analyses like shape analysis and motion planning without external dependencies.
Datasets
UniOcc unifies data from multiple sources:
- Real-world Datasets: nuScenes and Waymo.
- Synthetic Datasets: CARLA (for high-fidelity, virtually unlimited data) and OpenCOOD (for cooperative driving scenarios).
For the real-world datasets, UniOcc synthesizes occupancy ground truth using three pseudo-labeling pipelines: Occ3D, OpenOccupancy, and SurroundOcc. This multi-tool approach enhances the robustness and diversity of the generated labels.
Key Insights and Ideas
- Flow is Crucial for Forecasting: Explicitly modeling per-voxel flow (both dynamic and static) significantly improves occupancy forecasting performance and temporal consistency, especially for dynamic object classes.
- Domain Diversity is Key for Generalization: Training on a single dataset leads to poor out-of-distribution performance. UniOcc's unified dataset, combining real and synthetic data, enables models to generalize better to unseen domains and scenarios.
- Simulation Data Enhances Shape Learning: Synthetic data from CARLA, with its "perfect" object shapes, helps models learn more realistic object geometries, particularly improving predictions where LiDAR-based pseudo-labels are imperfect.
- Ground-Truth-Free Metrics are Essential: Standard IoU metrics can be misleading with imperfect labels. The proposed ground-truth-free metrics provide a more reliable assessment of model realism and temporal coherence, especially for generative tasks.
Key Results
- Flow Improves Performance: Augmenting OccWorld with voxel-level flow consistently improved forecasting performance and temporal consistency across various metrics.
- Unified Dataset Outperforms Single-Source Training: Models trained on the combined UniOcc dataset showed significantly better performance and generalization compared to models trained on individual datasets like nuScenes or Waymo.
- Simulation Data Improves Object Shape: Incorporating CARLA data led to more reasonable predicted car shapes, as illustrated by qualitative comparisons.
- Cooperative Perception Benefits: Integrating CoHFF demonstrated that multi-agent fusion within UniOcc can mitigate occlusions and improve coverage, yielding reasonable occupancy predictions.
Limitations and Future Work
- Sim-Real Compatibility: While simulation data improves object shape learning, a naive mix of simulation and real-world data can degrade temporal consistency. The paper notes a potential mismatch in speed ranges between CARLA and real-world datasets as a contributing factor. Future work could explore methods to better bridge this sim-real gap.
- Occupancy Prediction: While UniOcc primarily targets forecasting, it also facilitates camera-based occupancy prediction. The paper notes that integrating these models requires careful calibration and pre-processing.
- Future Research: The authors hope UniOcc will catalyze further research in occupancy-centric autonomous driving, fostering innovations in unified development, benchmarking, and evaluation. The framework's flexibility for multi-agent settings and its comprehensive evaluation metrics open avenues for exploring more advanced cooperative and generative occupancy tasks.
Summary updated