Depth Privileged Object Detection in Indoor Scenes via Deformation Hallucination

Zhijie Zhang, Yan Liu, Junjie Chen, Li Niu, Liqing Zhang

Abstract

Depth information provides critical geometric cues for indoor object detection, yet it is often unavailable during deployment. This work explores the Learning Using Privileged Information (LUPI) paradigm, where depth is used during training to improve RGB-only inference. We propose a novel approach that hallucinates geometric deformations—specifically sampling offsets for deformable convolutions—instead of raw depth maps. By distilling these offsets from a depth-based teacher to an RGB-based student, our model learns to focus on structural boundaries. Experiments on NYUDv2 and SUN RGB-D benchmarks demonstrate that our method consistently outperforms existing depth-hallucination techniques.

1. Introduction

In indoor environments, distinguishing objects from cluttered backgrounds or varying illumination is a significant challenge for standard RGB-based detectors. Depth data offers a "superpower" by providing geometric structure that is invariant to appearance. While depth sensors like LiDAR or Kinect can provide this data, they are often too expensive or impractical for widespread deployment.

Previous research in LUPI has attempted to "hallucinate" missing depth images or high-level depth features. However, these targets are often noisy or redundant. We argue that the most essential information provided by depth is geometric structure. Consequently, we propose to hallucinate the geometric deformations (sampling offsets) that allow a network to "see" the physical boundaries and shapes of objects.

2. Methodology

Our framework utilizes a three-branch architecture designed to transfer geometric knowledge from depth to RGB.

The proposed framework architecture featuring RGB (R), Hallucination (H), and Depth (D) branches.

2.1 The Three-Branch Architecture

  1. R-branch (RGB): A standard detection branch that processes RGB images to perform final classification and regression.
  2. D-branch (Depth): A "teacher" branch pre-trained on depth images (encoded as HHA). It learns optimal geometric deformations for object detection. This branch is frozen during the training of the other components.
  3. H-branch (Hallucination): A "student" branch that takes RGB input but is supervised to mimic the sampling offsets learned by the D-branch.

3. Geometric Deformation Hallucination

The core of our method lies in the use of Deformable Convolutions (DeformConv). Unlike standard convolutions that sample pixels in a rigid grid, DeformConv learns an "offset field" that allows the network to sample pixels based on the actual shape of the object.

Mechanism of deformable convolution showing the offset field and adaptive sampling.

By forcing the H-branch to predict the same offsets as the D-branch, we effectively teach the RGB model how to "warp" its filters to match physical geometry. To ensure high-quality knowledge transfer, we employ a Quality-Based Weighted Transfer Loss. This loss weights the distillation based on how much the depth-derived deformation actually improves the D-branch's detection performance, preventing the model from learning noisy or unhelpful offsets.

4. Experiments and Results

We evaluated our model on two primary indoor benchmarks: NYUDv2 (1,449 images) and SUN RGB-D (10,335 images). Both datasets were evaluated across 19 common object categories.

4.1 Quantitative Performance

Our method consistently outperforms the baseline and previous state-of-the-art (SOTA) hallucination methods like HallucinationNet and ROCK.

Method NYUDv2 (mAP) SUN RGB-D (mAP)
Baseline (RGB only) 42.73% 52.94%
HallucinationNet 45.22% 55.35%
ROCK 44.89% 55.14%
Ours (Full Method) 46.88% 56.84%

4.2 Key Insights

  • Structural Focus: Hallucinating deformations is more efficient than hallucinating raw pixels, as it focuses on the result of geometric understanding.
  • Color Invariance: The H-branch learns to identify structural cues (like edges and gaps) in RGB signals that correspond to physical boundaries, helping it overcome color-based clutter.
  • Positive Position Focus: Restricting hallucination to "positive" object positions reduces training noise and improves convergence.

5. Conclusion

We presented a novel framework for depth privileged object detection that leverages deformation hallucination. By distilling geometric sampling offsets from a depth teacher, our RGB-only model gains a structural understanding of indoor scenes. Future work will explore applying this "deformation distillation" to other vision tasks such as semantic and instance segmentation.

Summary updated

Abstract

RGB-D object detection has achieved significant advance, because depth provides complementary geometric information to RGB images. Considering depth images are unavailable in some scenarios, we focus on depth privileged object detection in indoor scenes, where the depth images are only available in the training phase. Under this setting, one prevalent research line is modality hallucination, in which depth image and depth feature are the common choices for hallucinating. In contrast, we choose to hallucinate depth deformation, which is explicit geometric information and efficient to hallucinate. Specifically, we employ the deformable convolution layer with augmented offsets as our deformation module and regard the offsets as geometric deformation, because the offsets enable flexibly sampling over the object and transforming to a canonical shape for ease of detection. In addition, we design a quality-based mechanism to avoid negative transfer of depth deformation. Experimental results and analyses on NYUDv2 and SUN RGB-D demonstrate the effectiveness of our method against the state-of-the-art methods for depth privileged object detection.
BibTeX
@inproceedings{zhang2021depth,
  author = {Zhijie Zhang and Yan Liu and Junjie Chen and Li Niu and Liqing Zhang},
  title = {{Depth Privileged Object Detection in Indoor Scenes via Deformation Hallucination}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16459},
}
Figure