Reducing Annotation Effort by Identifying and Labeling Contextually Diverse Classes for Semantic Segmentation Under Domain Shift

Sharat Agarwal, Saket Anand, Chetan AroraWACV 2023

TL;DR

This paper tackles the expensive problem of annotating data for semantic segmentation in domain adaptation by proposing a novel Active Domain Adaptation (ADA) strategy. Instead of annotating entire frames or random pixels, their method intelligently identifies and labels specific, contextually diverse classes within frames that are most challenging for the model, significantly reducing annotation effort while improving performance.

What Problem Does This Paper Address?

Semantic segmentation, a task crucial for understanding images (e.g., in autonomous driving), requires vast amounts of pixel-level annotated data. Acquiring this data is incredibly time-consuming and costly. Domain adaptation (DA) aims to leverage models trained on one domain (e.g., synthetic data) for another (e.g., real-world data), but a significant "domain shift" often leads to performance degradation. Active Domain Adaptation (ADA) tries to mitigate this by intelligently selecting which data to annotate from the target domain. However, existing ADA methods like MADA (which selects frames complementary to source anchors) or LabOR (which selects uncertain pixels) are either inefficient (wasting budget on redundant pixels) or tedious (labeling sparse, hard-to-annotate pixels). This paper addresses the need for a more cost-effective and efficient ADA strategy that focuses on annotating semantically meaningful, contextually diverse classes.

Methods

The core innovation lies in their "class-based annotation" strategy. Instead of annotating pixels or entire frames, they propose identifying and labeling specific classes within selected frames that are most informative for the model. They introduce two main approaches for this class selection:

  • Anchor-Based Class Selection: This method leverages class representative "anchors" derived from the currently labeled target data. For each unlabeled image, it computes the discrepancy between these class anchors and the model's predicted class confusion. Classes with a high discrepancy are deemed informative and selected for annotation. This aims to strengthen the model's representation of under-represented or confusing classes.
  • Augmentation-Based Class Selection: This approach utilizes self-supervised learning principles. It measures the disagreement in class confusion predictions between a model processing weakly augmented and strongly augmented versions of the same image. Higher disagreement suggests classes that are more sensitive to augmentations and thus harder for the model to learn, making them good candidates for annotation.

These class selection methods are designed to be used in conjunction with existing frame selection strategies (like CDAL). The overall pipeline involves an initial model warmup, active learning iterations where frames are selected and then informative classes within those frames are identified and labeled, followed by self-training using pseudo-labels from unlabeled data.

Datasets

The paper uses standard datasets for evaluating domain adaptation in semantic segmentation:

  • GTA5: A synthetic dataset with 24,966 images, used as the source domain.
  • Cityscapes: A real-world dataset with 2048x1024 resolution images, used as the target domain for evaluation.
  • Synthia: Another synthetic dataset with 9,400 images, also used as a source domain in conjunction with Cityscapes.

Key Insights and Ideas

  • Focusing on "Hard" Classes is Key: The central idea is that annotating specific, contextually difficult classes is more efficient than annotating entire frames or random pixels. This shifts the focus from pixel-level uncertainty to semantic-level difficulty.
  • Context Matters: The "hard" classes are not static; they are context-dependent and vary across frames. Their methods exploit this by analyzing class confusion within individual frames.
  • Class Anchors for Representation: The Anchor-based method's use of class anchors provides a robust way to represent learned class features and identify deviations, guiding the selection of informative classes.
  • Augmentation Sensitivity as a Signal: The Augmentation-based method cleverly uses the model's sensitivity to strong augmentations as a proxy for class difficulty, offering a data-driven way to pinpoint challenging regions.
  • Decorator Pattern for ADA: Their class selection methods can be integrated with existing frame selection strategies, acting as a "decorator" to improve their efficiency.

Key Results

  • Significant Performance Gains with Reduced Annotation: On the GTA5 → Cityscapes benchmark, their Anchor-based approach achieved 66.6 mIoU with only 4.7% annotation, outperforming MADA (64.9 mIoU with 5% annotation). The Aug-based approach also showed strong results.
  • Competitive with State-of-the-Art: They reduced the error margin (difference from fully supervised mIoU) significantly compared to other ADA techniques, achieving a 3-point improvement on GTA5→Cityscapes and a 2.5-point improvement on Synthia→Cityscapes, all with a minimal annotation budget (around 5%).
  • "Skyline" Performance: Their hypothetical IoU-based selection approach, which uses fully supervised information, achieved performance comparable to a fully supervised model with only ~7% of the data, validating the effectiveness of their class-selection paradigm.
  • Qualitative Improvements: Visualizations show more accurate and smoother segmentation masks in confusing regions compared to MADA.

Limitations and Future Work

  • Annotation Budget Control: The paper notes that controlling the exact annotation budget can be challenging with their class-selection approach, as it annotates entire classes rather than a fixed number of pixels.
  • Pseudo-Label Generation: While their method effectively uses pseudo-labels from unlabeled data, they suggest future work could focus on generating more informative pseudo-labels that are complementary to the actively labeled data.
  • Exploiting Source Data: They mention exploring how to better leverage abundant source labeled data in future work.
  • Source-Free Domain Adaptation: The paper demonstrates the utility of their approach in a Source-Free Domain Adaptation setting, showing promise for scenarios where source data is unavailable.

Summary updated

BibTeX
@inproceedings{agarwal2023reducing,
  author = {Sharat Agarwal and Saket Anand and Chetan Arora},
  title = {{Reducing Annotation Effort by Identifying and Labeling Contextually Diverse Classes for Semantic Segmentation Under Domain Shift}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure