Introduction
In the field of computer vision, dense prediction tasks—such as semantic segmentation, depth estimation, and surface normal estimation—are essential for scene understanding. While Multi-Task Learning (MTL) offers a way to solve these tasks simultaneously by leveraging shared representations, it faces a significant bottleneck: the scarcity of fully labeled datasets. In practice, most datasets are "partially supervised," meaning an image may have labels for depth but lack semantic segmentation masks.
The core challenge in Multi-Task Partially Supervised Learning (MTPSL) is capturing the relationships between different tasks when ground truth labels do not overlap. Previous methods often relied on global image-level vectors, which discard fine-grained spatial details, or pixel-to-pixel matching, which struggles to align discrete categorical data (like labels) with continuous numerical data (like depth). To bridge this gap, this paper introduces Region-aware Distribution Contrast (RDC), a framework that leverages structural priors and distributional modeling to align tasks effectively.
The RDC Framework
The RDC pipeline introduces three primary innovations designed to learn cross-task dependencies without requiring exhaustive labeling for every task.
1. SAM-Guided Region Extraction
To identify meaningful structures within an image without relying on task-specific labels, the framework utilizes the Segment Anything Model (SAM). SAM generates high-quality, class-agnostic region masks that serve as structural priors. These masks define the boundaries of objects and textures, providing a common spatial reference point for all tasks during the training phase.
2. Gaussian Distribution Modeling
Traditional methods often represent an image region using a single average feature vector. However, a single point in latent space cannot capture the complex internal texture and structural variations of a region. RDC instead models the features within each SAM-detected region as a Gaussian Distribution, calculating both the mean and the covariance. This approach allows the model to represent the "intra-region structure," which is crucial when trying to find commonalities between different types of data, such as the relationship between a "chair" category and its specific geometric depth profile.
3. Distributional Contrastive Learning
To align these regions across different tasks, the authors employ a contrastive learning strategy. The model is trained to pull the Gaussian distributions of the same region from different task branches (e.g., the segmentation branch and the depth branch) closer together, while pushing distributions from different regions apart.
To measure the distance between these distributions, the framework utilizes the Wasserstein distance. The researchers found that Wasserstein distance is significantly more robust than the standard Kullback-Leibler (KL) divergence for this application, as it better accounts for the underlying geometry of the feature space when task distributions do not perfectly overlap.
Experimental Results
The RDC framework was evaluated on two major dense prediction benchmarks: NYU-V2 (indoor scenes) and Cityscapes (urban driving scenes). The results demonstrate that RDC sets a new state-of-the-art (SOTA) for partially supervised learning:
- NYU-V2: In scenarios where labels were missing randomly, RDC improved semantic segmentation performance by 11.75% and surface normal estimation by a staggering 68.69% compared to previous SOTA methods.
- Cityscapes: The model achieved a 22.73% reduction in error for depth estimation compared to existing partially supervised approaches.
- Comparison to Full Supervision: Remarkably, RDC occasionally outperformed TaskExpert, a leading fully supervised method, particularly in surface normal estimation. This suggests that the distributional contrastive loss is highly effective at regularizing the model and capturing cross-task synergies.
Discussion and Future Work
A key advantage of the RDC framework is its efficiency during deployment. While the Segment Anything Model (SAM) is used to guide the learning process during training, it is not required during inference. This allows the multi-task network to maintain high-speed performance in real-world applications.
However, the method does have limitations. The training phase is computationally more intensive due to the inclusion of SAM and the calculation of covariance matrices for Gaussian modeling. Additionally, the model's performance is partially dependent on the quality of SAM’s zero-shot region proposals.
Future research may explore extending this distribution-based contrastive approach to multi-modal problems, such as aligning text descriptions with specific image regions, or applying it to video-based multi-task learning where temporal consistency is required.
Summary updated