Abstract
Depth information provides critical geometric cues for indoor object detection, yet it is often unavailable during deployment. This work explores the Learning Using Privileged Information (LUPI) paradigm, where depth is used during training to improve RGB-only inference. We propose a novel approach that hallucinates geometric deformations—specifically sampling offsets for deformable convolutions—instead of raw depth maps. By distilling these offsets from a depth-based teacher to an RGB-based student, our model learns to focus on structural boundaries. Experiments on NYUDv2 and SUN RGB-D benchmarks demonstrate that our method consistently outperforms existing depth-hallucination techniques.
1. Introduction
In indoor environments, distinguishing objects from cluttered backgrounds or varying illumination is a significant challenge for standard RGB-based detectors. Depth data offers a "superpower" by providing geometric structure that is invariant to appearance. While depth sensors like LiDAR or Kinect can provide this data, they are often too expensive or impractical for widespread deployment.
Previous research in LUPI has attempted to "hallucinate" missing depth images or high-level depth features. However, these targets are often noisy or redundant. We argue that the most essential information provided by depth is geometric structure. Consequently, we propose to hallucinate the geometric deformations (sampling offsets) that allow a network to "see" the physical boundaries and shapes of objects.
2. Methodology
Our framework utilizes a three-branch architecture designed to transfer geometric knowledge from depth to RGB.
2.1 The Three-Branch Architecture
- R-branch (RGB): A standard detection branch that processes RGB images to perform final classification and regression.
- D-branch (Depth): A "teacher" branch pre-trained on depth images (encoded as HHA). It learns optimal geometric deformations for object detection. This branch is frozen during the training of the other components.
- H-branch (Hallucination): A "student" branch that takes RGB input but is supervised to mimic the sampling offsets learned by the D-branch.
3. Geometric Deformation Hallucination
The core of our method lies in the use of Deformable Convolutions (DeformConv). Unlike standard convolutions that sample pixels in a rigid grid, DeformConv learns an "offset field" that allows the network to sample pixels based on the actual shape of the object.
By forcing the H-branch to predict the same offsets as the D-branch, we effectively teach the RGB model how to "warp" its filters to match physical geometry. To ensure high-quality knowledge transfer, we employ a Quality-Based Weighted Transfer Loss. This loss weights the distillation based on how much the depth-derived deformation actually improves the D-branch's detection performance, preventing the model from learning noisy or unhelpful offsets.
4. Experiments and Results
We evaluated our model on two primary indoor benchmarks: NYUDv2 (1,449 images) and SUN RGB-D (10,335 images). Both datasets were evaluated across 19 common object categories.
4.1 Quantitative Performance
Our method consistently outperforms the baseline and previous state-of-the-art (SOTA) hallucination methods like HallucinationNet and ROCK.
| Method | NYUDv2 (mAP) | SUN RGB-D (mAP) |
|---|---|---|
| Baseline (RGB only) | 42.73% | 52.94% |
| HallucinationNet | 45.22% | 55.35% |
| ROCK | 44.89% | 55.14% |
| Ours (Full Method) | 46.88% | 56.84% |
4.2 Key Insights
- Structural Focus: Hallucinating deformations is more efficient than hallucinating raw pixels, as it focuses on the result of geometric understanding.
- Color Invariance: The H-branch learns to identify structural cues (like edges and gaps) in RGB signals that correspond to physical boundaries, helping it overcome color-based clutter.
- Positive Position Focus: Restricting hallucination to "positive" object positions reduces training noise and improves convergence.
5. Conclusion
We presented a novel framework for depth privileged object detection that leverages deformation hallucination. By distilling geometric sampling offsets from a depth teacher, our RGB-only model gains a structural understanding of indoor scenes. Future work will explore applying this "deformation distillation" to other vision tasks such as semantic and instance segmentation.
Summary updated