Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction

Dubing Chen, Huan Zheng, Jin Fang, Xingping Dong, Xianfei Li, Wenlong Liao, Tao He, Pai Peng, Jianbing Shen

Introduction

In autonomous driving, vision-based 3D semantic occupancy prediction (VisionOcc) is critical for understanding the spatial environment. This task involves predicting which 3D voxels are occupied and identifying their semantic category (e.g., vehicle, drivable surface, or vegetation) using only camera inputs. While temporal information is essential for resolving ambiguities and handling dynamic objects, existing methods suffer from two primary limitations: a narrow focus on voxel-level feature stacking and prohibitive memory consumption as the historical sequence length increases.

The GDFusion Framework

GDFusion addresses these challenges by redefining temporal fusion through a Unified Gradient Descent View. The researchers demonstrate that standard Recurrent Neural Network (RNN) updates can be mathematically reinterpreted as a single step of gradient descent. This optimization-based perspective allows the model to minimize the discrepancy between current observations and historical states in a principled manner.

Overview of the three temporal cues integrated by GDFusion: Motion calibration, Geometric complementation, and Scene-level consistency.

By treating fusion as an optimization problem, GDFusion integrates three previously overlooked temporal cues into a single, streaming pipeline:

  1. Scene-Level Cue: This encodes global environmental properties, such as weather (rainy/sunny) or lighting (day/night), directly into the network's parameters. Using a self-supervised task, the model adapts these parameters frame-by-frame to maintain scene consistency.
  2. Temporal Motion Cue: To account for both ego-motion and the movement of dynamic objects, the model predicts a 3D flow map. This map aligns historical features with the current frame, ensuring that moving objects are correctly localized.
  3. Temporal Geometry Cue: By leveraging historical depth distributions, GDFusion "complements" the current frame's geometry. This makes the 2D-to-3D lifting process significantly more robust against occlusion and sensor noise.

Memory-Efficient Voxel Fusion

Unlike traditional multi-frame stacking methods (e.g., SOLOFusion) that require storing a growing queue of expensive 3D feature volumes, GDFusion maintains a single, compact hidden state. This state is updated via the RNN-as-optimization logic, allowing the model to benefit from long-term history without the linear memory growth typically associated with long sequences.

Experimental Results

GDFusion was evaluated on three major 3D occupancy benchmarks based on the nuScenes dataset: Occ3D, SurroundOcc, and OpenOccupancy. It serves as a "plug-and-play" enhancement for existing architectures like BEVDetOcc and FB-Occ.

Performance (mIoU) vs. FP32 Inference Memory. GDFusion variants (marked with -GF) achieve superior accuracy with significantly lower memory footprints compared to their respective baselines and competing methods.

Key Findings:

  • Accuracy Gains: On the Occ3D benchmark, GDFusion improved mean Intersection over Union (mIoU) by 1.4% to 4.8% across various baselines. The improvements were most notable in dynamic categories (pedestrians, cars) and background surfaces.
  • Efficiency: Compared to traditional stacking methods, GDFusion reduced memory consumption by 27% to 72%. This efficiency allows for higher-resolution predictions on standard hardware.
  • Speed: The inference overhead is negligible. In many cases, GDFusion outperforms stacking methods in terms of speed because it avoids processing multiple raw feature volumes simultaneously.

Conclusion

GDFusion represents a shift in how temporal information is handled in 3D perception. By unifying disparate cues—scene, motion, and geometry—under a gradient descent framework, it achieves a superior balance of accuracy and efficiency. Future work may explore extending this optimization view to more complex sensor modalities, such as LiDAR-camera fusion, or integrating long-sequence architectures like Mamba to further exploit temporal dependencies.

Summary updated

BibTeX
@inproceedings{chen2025rethinking,
  author = {Dubing Chen and Huan Zheng and Jin Fang and Xingping Dong and Xianfei Li and Wenlong Liao and Tao He and Pai Peng and Jianbing Shen},
  title = {{Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00148},
}
Figure