Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning

Meixuan Li, Tianyu Li, Guoqing Wang, Peng Wang, Yang Yang, Heng Tao ShenarXiv 2024 · DOI · arXiv

Introduction

In the field of computer vision, dense prediction tasks—such as semantic segmentation, depth estimation, and surface normal estimation—are essential for scene understanding. While Multi-Task Learning (MTL) offers a way to solve these tasks simultaneously by leveraging shared representations, it faces a significant bottleneck: the scarcity of fully labeled datasets. In practice, most datasets are "partially supervised," meaning an image may have labels for depth but lack semantic segmentation masks.

The core challenge in Multi-Task Partially Supervised Learning (MTPSL) is capturing the relationships between different tasks when ground truth labels do not overlap. Previous methods often relied on global image-level vectors, which discard fine-grained spatial details, or pixel-to-pixel matching, which struggles to align discrete categorical data (like labels) with continuous numerical data (like depth). To bridge this gap, this paper introduces Region-aware Distribution Contrast (RDC), a framework that leverages structural priors and distributional modeling to align tasks effectively.

The RDC Framework

The RDC pipeline introduces three primary innovations designed to learn cross-task dependencies without requiring exhaustive labeling for every task.

1. SAM-Guided Region Extraction

To identify meaningful structures within an image without relying on task-specific labels, the framework utilizes the Segment Anything Model (SAM). SAM generates high-quality, class-agnostic region masks that serve as structural priors. These masks define the boundaries of objects and textures, providing a common spatial reference point for all tasks during the training phase.

2. Gaussian Distribution Modeling

Traditional methods often represent an image region using a single average feature vector. However, a single point in latent space cannot capture the complex internal texture and structural variations of a region. RDC instead models the features within each SAM-detected region as a Gaussian Distribution, calculating both the mean and the covariance. This approach allows the model to represent the "intra-region structure," which is crucial when trying to find commonalities between different types of data, such as the relationship between a "chair" category and its specific geometric depth profile.

3. Distributional Contrastive Learning

To align these regions across different tasks, the authors employ a contrastive learning strategy. The model is trained to pull the Gaussian distributions of the same region from different task branches (e.g., the segmentation branch and the depth branch) closer together, while pushing distributions from different regions apart.

To measure the distance between these distributions, the framework utilizes the Wasserstein distance. The researchers found that Wasserstein distance is significantly more robust than the standard Kullback-Leibler (KL) divergence for this application, as it better accounts for the underlying geometry of the feature space when task distributions do not perfectly overlap.

Experimental Results

The RDC framework was evaluated on two major dense prediction benchmarks: NYU-V2 (indoor scenes) and Cityscapes (urban driving scenes). The results demonstrate that RDC sets a new state-of-the-art (SOTA) for partially supervised learning:

  • NYU-V2: In scenarios where labels were missing randomly, RDC improved semantic segmentation performance by 11.75% and surface normal estimation by a staggering 68.69% compared to previous SOTA methods.
  • Cityscapes: The model achieved a 22.73% reduction in error for depth estimation compared to existing partially supervised approaches.
  • Comparison to Full Supervision: Remarkably, RDC occasionally outperformed TaskExpert, a leading fully supervised method, particularly in surface normal estimation. This suggests that the distributional contrastive loss is highly effective at regularizing the model and capturing cross-task synergies.

Discussion and Future Work

A key advantage of the RDC framework is its efficiency during deployment. While the Segment Anything Model (SAM) is used to guide the learning process during training, it is not required during inference. This allows the multi-task network to maintain high-speed performance in real-world applications.

However, the method does have limitations. The training phase is computationally more intensive due to the inclusion of SAM and the calculation of covariance matrices for Gaussian modeling. Additionally, the model's performance is partially dependent on the quality of SAM’s zero-shot region proposals.

Future research may explore extending this distribution-based contrastive approach to multi-modal problems, such as aligning text descriptions with specific image regions, or applying it to video-based multi-task learning where temporal consistency is required.

Summary updated

Abstract

In this study, we address the intricate challenge of multi-task dense prediction, encompassing tasks such as semantic segmentation, depth estimation, and surface normal estimation, particularly when dealing with partially annotated data (MTPSL). The complexity arises from the absence of complete task labels for each training image. Given the inter-related nature of these pixel-wise dense tasks, our focus is on mining and capturing cross-task relationships. Existing solutions typically rely on learning global image representations for global cross-task image matching, imposing constraints that, unfortunately, sacrifice the finer structures within the images. Attempting local matching as a remedy faces hurdles due to the lack of precise region supervision, making local alignment a challenging endeavor. The introduction of Segment Anything Model (SAM) sheds light on addressing local alignment challenges by providing free and high-quality solutions for region detection. Leveraging SAM-detected regions, the subsequent challenge lies in aligning the representations within these regions. Diverging from conventional methods that directly learn a monolithic image representation, our proposal involves modeling region-wise representations using Gaussian Distributions. Aligning these distributions between corresponding regions from different tasks imparts higher flexibility and capacity to capture intra-region structures, accommodating a broader range of tasks. This innovative approach significantly enhances our ability to effectively capture cross-task relationships, resulting in improved overall performance in partially supervised multi-task dense prediction scenarios. Extensive experiments conducted on two widely used benchmarks underscore the superior effectiveness of our proposed method, showcasing state-of-the-art performance even when compared to fully supervised methods.
BibTeX
@misc{li2024region,
  author = {Meixuan Li and Tianyu Li and Guoqing Wang and Peng Wang and Yang Yang and Heng Tao Shen},
  title = {{Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning}},
  year = {2024},
  eprint = {2403.10252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10252},
}
Figure