Introduction
In autonomous driving, vision-based 3D semantic occupancy prediction (VisionOcc) is critical for understanding the spatial environment. This task involves predicting which 3D voxels are occupied and identifying their semantic category (e.g., vehicle, drivable surface, or vegetation) using only camera inputs. While temporal information is essential for resolving ambiguities and handling dynamic objects, existing methods suffer from two primary limitations: a narrow focus on voxel-level feature stacking and prohibitive memory consumption as the historical sequence length increases.
The GDFusion Framework
GDFusion addresses these challenges by redefining temporal fusion through a Unified Gradient Descent View. The researchers demonstrate that standard Recurrent Neural Network (RNN) updates can be mathematically reinterpreted as a single step of gradient descent. This optimization-based perspective allows the model to minimize the discrepancy between current observations and historical states in a principled manner.
By treating fusion as an optimization problem, GDFusion integrates three previously overlooked temporal cues into a single, streaming pipeline:
- Scene-Level Cue: This encodes global environmental properties, such as weather (rainy/sunny) or lighting (day/night), directly into the network's parameters. Using a self-supervised task, the model adapts these parameters frame-by-frame to maintain scene consistency.
- Temporal Motion Cue: To account for both ego-motion and the movement of dynamic objects, the model predicts a 3D flow map. This map aligns historical features with the current frame, ensuring that moving objects are correctly localized.
- Temporal Geometry Cue: By leveraging historical depth distributions, GDFusion "complements" the current frame's geometry. This makes the 2D-to-3D lifting process significantly more robust against occlusion and sensor noise.
Memory-Efficient Voxel Fusion
Unlike traditional multi-frame stacking methods (e.g., SOLOFusion) that require storing a growing queue of expensive 3D feature volumes, GDFusion maintains a single, compact hidden state. This state is updated via the RNN-as-optimization logic, allowing the model to benefit from long-term history without the linear memory growth typically associated with long sequences.
Experimental Results
GDFusion was evaluated on three major 3D occupancy benchmarks based on the nuScenes dataset: Occ3D, SurroundOcc, and OpenOccupancy. It serves as a "plug-and-play" enhancement for existing architectures like BEVDetOcc and FB-Occ.
Key Findings:
- Accuracy Gains: On the Occ3D benchmark, GDFusion improved mean Intersection over Union (mIoU) by 1.4% to 4.8% across various baselines. The improvements were most notable in dynamic categories (pedestrians, cars) and background surfaces.
- Efficiency: Compared to traditional stacking methods, GDFusion reduced memory consumption by 27% to 72%. This efficiency allows for higher-resolution predictions on standard hardware.
- Speed: The inference overhead is negligible. In many cases, GDFusion outperforms stacking methods in terms of speed because it avoids processing multiple raw feature volumes simultaneously.
Conclusion
GDFusion represents a shift in how temporal information is handled in 3D perception. By unifying disparate cues—scene, motion, and geometry—under a gradient descent framework, it achieves a superior balance of accuracy and efficiency. Future work may explore extending this optimization view to more complex sensor modalities, such as LiDAR-camera fusion, or integrating long-sequence architectures like Mamba to further exploit temporal dependencies.
Summary updated