Abstract
This paper introduces a modular Joint Refinement Network (JRN) that combines state-of-the-art depth estimation and semantic segmentation to improve both tasks simultaneously. Beyond boosting accuracy, the authors propose a novel way to quantify "cross-modality influence," revealing that the relationship between task synergy and performance is a delicate balance rather than a simple linear improvement.
Introduction
In computer vision, "holistic scene understanding" requires solving multiple tasks simultaneously, such as identifying object semantics and their 3D depth. Historically, research has diverged into two paths: highly optimized single-task models that ignore cross-task correlations, and monolithic joint models that act as "black boxes," making them difficult to train and debug.
This work addresses the challenge of creating a modular joint architecture. The goal is to combine specialized models so they mutually refine each other without increasing system complexity. Crucially, the authors seek to quantify the specific influence one task (e.g., semantics) exerts on another (e.g., depth) during the refinement process.
Methodology
The Joint Refinement Network (JRN)
The authors propose a two-stage modular pipeline. In the first stage, existing state-of-the-art (SOTA) "specialist" models generate initial depth maps and semantic labels from a single RGB image. In the second stage, the Joint Refinement Network (JRN) takes these initial "soft" predictions as input.
The JRN employs a multi-scale architecture, processing inputs at 1/8, 1/4, and 1/2 resolutions to capture both global context and fine geometric details. Several fusion strategies were evaluated, including:
- Concatenation (Cat60): A loose coupling where features are stacked.
- Summation (Sum60): A coupled computation forcing a joint feature representation.
- Channel Squeezing: Reducing feature channels (Cat10, Cat5, Cat1) to force information compression and increase inter-modality reliance.
Quantifying Cross-Modality Influence
To measure synergy, the authors introduced an "influence number" ($\omega$). This metric is calculated by "muting" one input (e.g., providing only the depth map) and comparing the resulting output to the joint prediction.
By mathematically isolating the contribution of the missing modality, the researchers can determine how much the interaction between tasks actually drives performance gains.
Experimental Results and Discussion
The study utilized the NYU-Depth v2 benchmark, consisting of 1,449 indoor RGB-D images. Semantic labels were mapped into five major geometric categories: Ground, Vertical, Ceiling, Furniture, and Object.
Performance Gains
The JRN successfully improved upon its SOTA inputs:
- Depth Estimation: The Sum60 variant achieved a squared relative error of 0.123, outperforming the input model (0.125) and significantly exceeding previous joint models like Joint HCRF (0.210).
- Semantic Segmentation: Mean Intersection over Union (mIOU) improved from 53.28% (baseline FCN) to 54.27% (Sum60).
- Qualitative Improvements: Visually, the JRN recovered sharper depth edges (e.g., bed frames) and corrected inconsistent semantic labels (e.g., removing "ground" pixels from furniture).
Key Insights
- The "Sweet Spot" of Influence: The relationship between cross-modality influence and accuracy is non-linear and concave. While moderate influence is beneficial—such as a chair's semantic shape refining its depth edges—excessive influence can be detrimental.
- Negative Influence: If influence is too strong, errors in one task can "leak" into the other, creating artifacts like noisy semantic boundaries causing "halos" in depth maps.
- Coupled vs. Loose Fusion: Summation (coupled) generally outperformed concatenation (loose), suggesting that forcing a shared representation leads to more robust refinement.
Conclusion
The modular JRN demonstrates that joint refinement can significantly improve scene understanding while remaining flexible. Unlike monolithic models, this approach allows for easier training with heterogeneous data and the simple addition of new modalities. Future work will explore end-to-end training and scaling the architecture to include additional tasks like surface normals or reflectance.
Summary updated