LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-Based 3D Semantic Occupancy Prediction

Linqing Zhao, Xiuwei Xu, Ziwei Wang, Yunpeng Zhang, Borui Zhang, Wenzhao Zheng, Dalong Du, Jie Zhou, Jiwen Lu

Introduction

3D semantic occupancy prediction is a critical task for autonomous driving, requiring models to predict both the geometry and semantics of a 3D scene from 2D camera inputs. Traditional methods rely on dense 3D voxel grids, which are computationally expensive and highly redundant; in datasets like nuScenes, over 95% of voxels are empty. This sparsity leads to significant overhead and a high risk of overfitting to noise in empty space. LowRankOcc addresses these challenges by treating 3D voxel grids as low-rank tensors, factorizing dense data into compact 1D and 2D components to capture essential scene geometry efficiently while maintaining high performance.

Methodology: Decomposition and Recovery

LowRankOcc utilizes a "decomposition-encoding-recovery" framework designed to exploit the intrinsic low-rank properties of outdoor driving environments. By moving the heavy lifting from 3D space to 1D and 2D spaces, the model increases its receptive field much more cheaply than traditional 3D methods.

Vertical-Horizontal (VH) Decomposition

The core innovation is the factorization of the 3D tensor into a vertical vector (Z-axis) and a horizontal matrix (XY-plane). This reduces the computational complexity from $O(XYZ)$ to $O(R(XY + Z))$, where $R$ represents the rank. This approach is particularly effective because the vertical dimension in driving scenes often contains significant redundancy, allowing the model to focus its capacity on the more complex horizontal spatial relationships.

Recursive Residual Decomposition (RRD)

To balance global structure with fine-grained detail, the model employs a recursive strategy. An initial decomposition block captures low-frequency information, such as ground planes and large building layouts. Subsequent blocks then process the "residual"---the difference between the original signal and the current reconstruction---to recover high-frequency details like pedestrians or thin poles. This ensures that the low-rank approximation does not sacrifice the precision needed for safe navigation.

Illustration of the low-rank decomposition modes and the trade-off between recovery time and loss.

Parallel Feature Encoding and Recovery

The decomposed components are grouped into mini-batches and processed in parallel using efficient 1D and 2D convolutions or attention mechanisms. This parallelization bypasses the need for heavy, resource-intensive 3D convolutions. Finally, an outer product operation reconstructs these processed low-rank features back into a 3D representation for the final occupancy prediction head.

Key Insights

  • Z-Axis Redundancy: Prioritizing decomposition along the vertical axis yields a superior trade-off between speed and accuracy compared to standard Canonical-Polyadic (CP) decomposition.
  • Residual Learning for Tensors: Just as ResNets revolutionized deep learning by learning residuals, this paper shows that decomposing residuals allows a low-rank model to gradually "fill in" the details that a single-pass decomposition might miss.
  • Efficiency via Dimensionality: Shifting computation to lower-dimensional spaces allows for a larger receptive field, enabling the model to "see" more of the scene at once without the quadratic or cubic cost increases associated with 3D grids.

Experimental Results

The model was evaluated on the SemanticKITTI (monocular) and nuScenes (multi-camera) benchmarks, using dense annotations for rigorous testing.

  • SemanticKITTI: LowRankOcc achieved state-of-the-art (SOTA) performance with 38.47 SC IoU and 13.56 SSC mIoU, significantly outperforming previous leaders like MonoScene and OccFormer.
  • nuScenes: The model reached 32.78 SC IoU and 21.51 SSC mIoU, representing a roughly 1% absolute improvement over previous SOTA methods while demonstrating greater robustness to noise.
  • Qualitative Performance: Visual results show that LowRankOcc is much better at predicting intricate structures, such as distant crossroads and fine-grained object shapes, even in regions not directly visible in the input images.

Conclusion and Future Work

LowRankOcc demonstrates that low-rank tensor recovery is a powerful paradigm for efficient 3D occupancy prediction. By leveraging the inherent structure of driving scenes, it achieves SOTA results with reduced computational overhead. Future work includes automating rank selection for different scenes, extending the framework to temporal data for dynamic scenes, and further optimizing implementation for specific hardware to fully realize the theoretical latency gains.

Summary updated

Abstract

In this paper, we present a tensor decomposition and low-rank recovery approach (LowRankOcc) for vision-based 3D semantic occupancy prediction. Conventional methods model outdoor scenes with fine-grained 3D grids, but the sparsity of non-empty voxels introduces consider-able spatial redundancy, leading to potential overfitting risks. In contrast, our approach leverages the intrinsic low-rank property of 3D occupancy data, factorizing voxel representations into low-rank components to efficiently mitigate spatial redundancy without sacrificing performance. Specifically, we present the Vertical-Horizontal (VH) de-composition block factorizes 3D tensors into vertical vectors and horizontal matrices. With our “decomposition-encoding-recovery” framework, we encode 3D contexts with only 1/2D convolutions and poolings, and subsequently recover the encoded compact yet informative context features back to voxel representations. Experimental results demonstrate that LowRankOcc achieves state-of-the-art performances in semantic scene completion on the Se-manticKITTI dataset and 3D occupancy prediction on the nuScenes dataset.
BibTeX
@inproceedings{zhao2024lowrankocc,
  author = {Linqing Zhao and Xiuwei Xu and Ziwei Wang and Yunpeng Zhang and Borui Zhang and Wenzhao Zheng and Dalong Du and Jie Zhou and Jiwen Lu},
  title = {{LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-Based 3D Semantic Occupancy Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00936},
}
Figure