CGOcc: Learning Cross-Gaussian Dependencies for 3D Semantic Occupancy Prediction

Zirui Huang, Jiongle He, Yunfan Zeng, Yuanzhi Lu, Guangye Li, Boyan Tang, Weite Su, Pedro V. Sander, Amy Hui ZhangarXiv 2026 · arXiv

TL;DR

CGOcc is a novel framework for 3D semantic occupancy prediction that explicitly models the dependencies between overlapping 3D Gaussian primitives. By moving beyond simple weighted averages to a "dependency-aware" readout, it achieves state-of-the-art performance on the nuScenes benchmark, particularly excelling in complex regions where multiple scene representations overlap.

What Problem Does This Paper Address?

In autonomous driving, understanding the 3D world means more than just drawing boxes around cars; it requires 3D semantic occupancy prediction—predicting which parts of 3D space are occupied and by what (e.g., road, vehicle, vegetation).

Recent high-performing methods represent scenes using "semantic Gaussians"—adaptive, continuous primitives. However, these Gaussians often overlap. When a model needs to decide the final classification for a specific 3D point (voxel), current methods typically use a spatially weighted mixture of the overlapping Gaussians' individual predictions.

The authors argue this is insufficient. A simple mixture ignores the relational evidence between primitives. For example, if two overlapping Gaussians both suggest "vehicle," their joint interaction should provide stronger evidence than if they were considered independently. Without explicit modeling of these cross-primitive dependencies, the model struggles with ambiguity, especially near object boundaries and in dense, overlapping regions.

A schematic vehicle–road example compares weighted aggregation with CGOcc. Note how CGOcc uses cross-Gaussian dependencies (pale yellow edges) to resolve ambiguity.

Methods

CGOcc (Cross-Gaussian Occupancy) introduces a supervised multi-camera framework with two primary innovations:

  1. Geometry-Conditioned Multi-Source Interaction: The model initializes Gaussian centers using metric depth (from Metric3D v2) and refines them by integrating heterogeneous observations. It pulls appearance features from DINOv2 and geometric features from VGGT. These are fused using geometry-biased cross-attention and local graph attention, allowing Gaussians to "talk" to their neighbors and incorporate local spatial context.
  2. Dependency-Aware Semantic Prediction: This is the core of the paper. Instead of a simple weighted sum, CGOcc models dependencies using a diagonal-plus-low-rank representation.
  3. Efficiency: Rather than constructing a massive, computationally expensive $N \times N$ dependency matrix for all $N$ Gaussians, they use a factorized approach.
  4. Class-Conditional Projection: It extracts diagonal and signed cross-Gaussian statistics specific to each semantic class.
  5. Mass-Preserving Readout: These statistics refine the final voxel probabilities. Crucially, this refinement is "mass-preserving," meaning it redistributes probability among occupied classes without changing the overall estimate of whether a voxel is occupied or free.
Figure 2. Overview of CGOcc. Depth-initialized Gaussians undergo two geometry-conditioned multi-source interaction blocks before dependency-aware semantic readout (Fig. 3). Only Block 1 uses local graph attention; snowflakes denote frozen modules.

Datasets

The authors evaluated CGOcc on two major benchmarks derived from the nuScenes dataset:

  • Occ3D-nuScenes: A large-scale benchmark with 17 occupied semantic classes and dense annotations.
  • SurroundOcc-nuScenes: Used for additional validation to ensure the method's robustness across different evaluation protocols.

Key Insights and Ideas

  • Relational Evidence is Discriminative: The "aha moment" is that the interaction between primitives is just as important as the primitives themselves. Modeling these as a covariance-like dependency allows the model to "clean up" ambiguous regions.
  • Low-Rank Factorization is Key for Scaling: Modeling dependencies between thousands of Gaussians is normally a quadratic nightmare. By using a low-rank factor ($\ell_g$), the authors make cross-Gaussian modeling computationally feasible for real-time-adjacent applications.
  • Synergy with Interaction: Ablation studies showed that better feature interaction in the decoder (like neighbor message passing) actually makes the dependency-aware readout more effective. The two modules work together: one builds better features, the other uses them to model complex relationships.

Key Results

CGOcc achieves state-of-the-art results on the Occ3D-nuScenes validation set:

  • mIoU: 31.10%, outperforming the previous leader, GaussianFormer-2 (29.62%).
  • OccIoU: 56.32%, a significant jump over GaussianFormer-2 (53.75%).
  • Spatial Analysis: The performance gains are most pronounced in regions with high Gaussian overlap and near semantic boundaries (roughly 0.4m from ground-truth interfaces), proving that the dependency modeling is doing exactly what it was designed for.

Qualitatively, CGOcc produces much "cleaner" occupancy maps with more continuous drivable surfaces and fewer scattered, noisy predictions compared to baseline methods.

Qualitative comparison showing CGOcc's superior ability to predict continuous surfaces and regular vehicle shapes.

Limitations and Future Work

Despite its strengths, CGOcc has a few acknowledged limitations:

  • External Dependencies: It relies heavily on pre-calibrated cameras and depth estimates from a pretrained foundation model (Metric3D). Errors in initial depth can propagate through the system.
  • Temporal Modeling: The current version operates on individual frames. It does not explicitly model how these Gaussian dependencies evolve over time across a video sequence.
  • Future Directions: The authors suggest that future work could focus on making the system more robust to depth initialization errors and extending the dependency modeling to work across temporal frames for even smoother predictions.

Summary updated

Abstract

In Gaussian-based semantic occupancy prediction, overlapping primitives can provide ambiguous class predictions, making their joint interpretation important for voxel classification. However, current methods using spatially weighted mixtures neglect explicit modeling of cross-primitive dependencies, limiting the readout's ability to utilize category-specific relational evidence. Hence, we introduce CGOcc, a supervised multi-camera framework that explicitly models Cross-Gaussian dependencies for 3D semantic Occupancy prediction. CGOcc integrates appearance and geometric observations to refine Gaussian states and encodes cross-Gaussian dependencies in a diagonal-plus-low-rank representation. To incorporate these dependencies into voxel classification, CGOcc uses a class-conditional projection to extract diagonal and signed cross-Gaussian statistics without constructing a dense dependency matrix. The readout then refines occupied-class probabilities using these statistics while preserving a learned reference's occupied mass and free probability. Extensive experiments on nuScenes demonstrate CGOcc achieves state-of-the-art performance, with larger gains at high Gaussian overlap and improved predictions near boundaries. Ablation studies validate the effectiveness of covariance-aware prediction and support its synergy with geometric interaction.
BibTeX
@misc{huang2026cgocc,
  author = {Zirui Huang and Jiongle He and Yunfan Zeng and Yuanzhi Lu and Guangye Li and Boyan Tang and Weite SU and Pedro V. Sander and Amy Hui Zhang},
  title = {{CGOcc: Learning Cross-Gaussian Dependencies for 3D Semantic Occupancy Prediction}},
  year = {2026},
  eprint = {2611.15692},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15692},
}

Loading related content...

Figure