Introduction
3D semantic occupancy prediction is a critical task for autonomous driving, requiring models to predict both the geometry and semantics of a 3D scene from 2D camera inputs. Traditional methods rely on dense 3D voxel grids, which are computationally expensive and highly redundant; in datasets like nuScenes, over 95% of voxels are empty. This sparsity leads to significant overhead and a high risk of overfitting to noise in empty space. LowRankOcc addresses these challenges by treating 3D voxel grids as low-rank tensors, factorizing dense data into compact 1D and 2D components to capture essential scene geometry efficiently while maintaining high performance.
Methodology: Decomposition and Recovery
LowRankOcc utilizes a "decomposition-encoding-recovery" framework designed to exploit the intrinsic low-rank properties of outdoor driving environments. By moving the heavy lifting from 3D space to 1D and 2D spaces, the model increases its receptive field much more cheaply than traditional 3D methods.
Vertical-Horizontal (VH) Decomposition
The core innovation is the factorization of the 3D tensor into a vertical vector (Z-axis) and a horizontal matrix (XY-plane). This reduces the computational complexity from $O(XYZ)$ to $O(R(XY + Z))$, where $R$ represents the rank. This approach is particularly effective because the vertical dimension in driving scenes often contains significant redundancy, allowing the model to focus its capacity on the more complex horizontal spatial relationships.
Recursive Residual Decomposition (RRD)
To balance global structure with fine-grained detail, the model employs a recursive strategy. An initial decomposition block captures low-frequency information, such as ground planes and large building layouts. Subsequent blocks then process the "residual"---the difference between the original signal and the current reconstruction---to recover high-frequency details like pedestrians or thin poles. This ensures that the low-rank approximation does not sacrifice the precision needed for safe navigation.
Parallel Feature Encoding and Recovery
The decomposed components are grouped into mini-batches and processed in parallel using efficient 1D and 2D convolutions or attention mechanisms. This parallelization bypasses the need for heavy, resource-intensive 3D convolutions. Finally, an outer product operation reconstructs these processed low-rank features back into a 3D representation for the final occupancy prediction head.
Key Insights
- Z-Axis Redundancy: Prioritizing decomposition along the vertical axis yields a superior trade-off between speed and accuracy compared to standard Canonical-Polyadic (CP) decomposition.
- Residual Learning for Tensors: Just as ResNets revolutionized deep learning by learning residuals, this paper shows that decomposing residuals allows a low-rank model to gradually "fill in" the details that a single-pass decomposition might miss.
- Efficiency via Dimensionality: Shifting computation to lower-dimensional spaces allows for a larger receptive field, enabling the model to "see" more of the scene at once without the quadratic or cubic cost increases associated with 3D grids.
Experimental Results
The model was evaluated on the SemanticKITTI (monocular) and nuScenes (multi-camera) benchmarks, using dense annotations for rigorous testing.
- SemanticKITTI: LowRankOcc achieved state-of-the-art (SOTA) performance with 38.47 SC IoU and 13.56 SSC mIoU, significantly outperforming previous leaders like MonoScene and OccFormer.
- nuScenes: The model reached 32.78 SC IoU and 21.51 SSC mIoU, representing a roughly 1% absolute improvement over previous SOTA methods while demonstrating greater robustness to noise.
- Qualitative Performance: Visual results show that LowRankOcc is much better at predicting intricate structures, such as distant crossroads and fine-grained object shapes, even in regions not directly visible in the input images.
Conclusion and Future Work
LowRankOcc demonstrates that low-rank tensor recovery is a powerful paradigm for efficient 3D occupancy prediction. By leveraging the inherent structure of driving scenes, it achieves SOTA results with reduced computational overhead. Future work includes automating rank selection for different scenes, extending the framework to temporal data for dynamic scenes, and further optimizing implementation for specific hardware to fully realize the theoretical latency gains.
Summary updated