Probabilistic Indoor 3D Object Detection from RGB-D via Gaussian Distribution Estimation

H. Jin KimMathematics 2026 · DOI

TL;DR

This paper introduces a novel framework for 3D object detection using RGB-D data that represents objects as probabilistic 3D Gaussian distributions instead of traditional bounding boxes. This approach, built on a transformer architecture, allows for the explicit modeling of object uncertainty and achieves competitive performance on the SUN RGB-D benchmark, outperforming many existing methods.

What Problem Does This Paper Address?

Traditional 3D object detection methods often rely on deterministic bounding boxes to represent objects. This approach has limitations: it fails to capture the inherent uncertainty in object appearance, geometric projection, and the 3D structure of a scene. While some recent work has explored modeling objects as probability distributions (like Gaussians), these methods have typically been limited to 2D images or required complex late fusion of RGB and depth information, leading to a "modality gap." This paper aims to bridge this gap by proposing a unified, end-to-end framework for direct 3D object detection from RGB-D input that leverages probabilistic representations and early fusion within a transformer architecture.

Methods

The core of this work is a unified Gaussian-based framework for direct 3D object detection from RGB-D inputs. The key methodological contributions are:

  • Probabilistic 3D Gaussian Representation: Instead of predicting bounding box parameters, the network predicts the mean and covariance of a 3D Gaussian distribution for each object. This allows for the representation of both object location and geometric uncertainty as a 3D ellipsoid.
  • Unified RGB-D Input and Early Fusion: The model takes a four-channel RGB-D tensor as input. This is achieved by modifying a pre-trained vision transformer's patch embedding layer to accept both RGB and depth information from the outset. This early fusion approach aims to reduce the modality gap and enable more coherent feature learning.
  • Transformer Backbone: A vision transformer (ViT-Small) is used as the backbone to capture global context. The patch embedding layer is specifically adapted to handle the four-channel RGB-D input.
  • Cholesky-Based Covariance Parameterization: To ensure positive definiteness and stability during training, the covariance matrices are parameterized using a Cholesky factorization. This allows the network to output valid covariance estimates.
  • Distributional Loss Functions: The training objective uses divergence metrics like Kullback–Leibler (KL) and Bhattacharyya distances between predicted and ground-truth Gaussian distributions. This provides a geometrically meaningful training signal that aligns well with overlap metrics.
  • Geometric Transformations: The paper details how image-aligned Gaussian predictions are transformed into metrically interpretable 3D ellipsoids in the camera frame, accounting for perspective projection and depth-dependent scaling.

Datasets

The proposed method is evaluated on the SUN RGB-D benchmark. This dataset is widely used for indoor 3D object detection and contains 10,335 RGB-D images collected from various depth sensors across diverse indoor environments. The benchmark presents challenges such as cluttered scenes, significant inter-class variation, partial occlusions, and depth noise. The authors follow the official split for training and testing.

Key Insights and Ideas

  • Uncertainty is Key: Representing objects as probabilistic distributions (Gaussians) naturally encodes localization and shape uncertainty, which is crucial for real-world applications where perfect knowledge is rare.
  • Early Fusion Wins for Geometry: Integrating RGB and depth information at the earliest stage (via a four-channel tensor and transformer) is more effective for dense geometric tasks like 3D object detection than late fusion strategies. This allows the network to learn joint appearance-geometry correlations from the start.
  • Transformers Excel at Global Context: The self-attention mechanism in transformers is vital for capturing the global context needed to accurately estimate 3D Gaussian covariances, which can span large spatial extents and are influenced by surrounding scene elements.
  • Probabilistic Representation for Stability: The continuous and differentiable nature of Gaussian parameterization, coupled with distributional loss functions, leads to smoother gradient flow and more stable optimization compared to traditional box regression.
  • Covariance Matters: Modeling full cross-axis correlations in the covariance matrix (using Cholesky factorization) significantly improves performance over using only diagonal covariances, highlighting the importance of capturing object orientation and shape anisotropy.

Key Results

  • Competitive Performance: The proposed method achieves an [email protected] of 61.9% on the SUN RGB-D benchmark.
  • Outperforms Point-Cloud Methods: It surpasses several state-of-the-art point-cloud detectors like VoteNet (59.1%), H3DNet (60.1%), MLCVNet (59.8%), and 3DETR (59.1%).
  • Strong RGB-D Performance: The method significantly outperforms traditional RGB-D fusion strategies such as Frustum PointNets (54.0%), PointFusion (45.4%), and DSS (42.1%).
  • Second-Best RGB-D: It is the second-best RGB-D method on the benchmark, closely following ImVoteNet (63.4%), but with a simpler, single-stage architecture and inherent uncertainty quantification.
  • Qualitative Improvements: The Gaussian ellipsoids provide more geometrically interpretable detections, naturally capturing object shapes and orientations (e.g., stretching along desk edges). The model also gracefully handles uncertainty by increasing ellipsoid size when confidence is low, offering a more safety-aware output.
  • Ablation Studies: Replacing bounding box regression with a Gaussian head yields a +4.1 mAP improvement. Using full Cholesky parameterization for covariance provides a +3.2 mAP gain over diagonal covariance.

Limitations and Future Work

  • Single-Frame Assumption: The current formulation operates on single frames, limiting its ability to leverage temporal information from videos.
  • Single Ellipsoid Representation: Each object is represented by a single Gaussian ellipsoid, which might not be sufficient for highly complex objects with multiple distinct parts.
  • Computational Cost: The transformer backbone contributes to computational cost, potentially limiting real-time deployment.
  • Future Work Suggestions:
    • Temporal modeling for video data.
  • Mixture-of-Gaussians for representing more complex objects.
  • Lightweight backbone designs for real-time applications.
  • Leveraging uncertainty for downstream tasks like active perception.
  • Investigating the impact of different depth-dependent scaling approximations.

Summary updated

Abstract

Conventional object detectors represent each object by a deterministic bounding box, regressing its center and size from RGB images. However, such discrete parameterization ignores the inherent uncertainty in object appearance and geometric projection, which can be more naturally modeled as a probabilistic density field. Recent works have introduced Gaussian-based formulations that treat objects as distributions rather than boxes, yet they remain limited to 2D images or require late fusion between image and depth modalities. In this paper, we propose a unified Gaussian-based framework for direct 3D object detection from RGB-D inputs. Our method is built upon a vision transformer backbone to effectively capture global context. Instead of separately embedding RGB and depth features or refining depth within region proposals, our method takes a full four-channel RGB-D tensor and predicts the mean and covariance of a 3D Gaussian distribution for each object in a single forward pass. We extend a pretrained vision transformer to accept four-channel inputs by augmenting the patch embedding layer while preserving ImageNet-learned representations. This formulation allows the detector to represent both object location and geometric uncertainty in 3D space. By optimizing divergence metrics such as the Kullback–Leibler or Bhattacharyya distances between predicted and target distributions, the network learns a physically consistent probabilistic representation of objects. Experimental results on the SUN RGB-D benchmark demonstrate that our approach achieves competitive performance compared to state-of-the-art point-cloud-based methods while offering uncertainty-aware and geometrically interpretable 3D detections.
BibTeX
@inproceedings{kim2026probabilistic,
  author = {H. Jin Kim},
  title = {{Probabilistic Indoor 3D Object Detection from RGB-D via Gaussian Distribution Estimation}},
  booktitle = {Mathematics},
  year = {2026},
  doi = {10.3390/math14030421},
}
Figure