PICA: Point-Wise Instance and Centroid Alignment Based Few-Shot Domain Adaptive Object Detection With Loose Annotations

Chaoliang Zhong, Jie Wang, Cheng Feng, Ying Zhang, Jun Sun, Yasuto YokotaWACV 2022

TL;DR

This paper introduces PICA, a novel method for few-shot domain adaptive object detection that tackles the challenge of "loose annotations" (only a few objects annotated per image). PICA extends point-wise alignment techniques to object detection and introduces a moving average centroid mechanism to robustly handle noisy background labels and scarce target data, achieving state-of-the-art results.

What Problem Does This Paper Address?

Training accurate object detection models typically requires a massive amount of meticulously annotated data. This annotation process is expensive and time-consuming. Domain adaptation aims to bridge the gap between a source domain with abundant labeled data and a target domain with limited or no labels. However, many existing domain adaptation methods either require large amounts of unlabeled target data (unsupervised domain adaptation - UDA) or struggle with scenarios where target data is not only scarce but also loosely annotated. Loose annotation means that in target images, only a handful of objects are annotated, leaving many other objects (and thus, background regions) potentially mislabeled. This "label noise" in background regions, combined with the scarcity of annotated target instances, poses a significant challenge for existing instance-level alignment methods, which are crucial for adapting models to new domains. This paper specifically addresses this "few-shot loose annotation" setting for object detection.

Methods

PICA builds upon the Faster R-CNN framework and integrates three key alignment strategies:

  • Global Alignment: Similar to existing UDA methods, PICA uses a global discriminator to align image-level features between source and target domains, aiming for domain-invariant representations.
  • Local Alignment: A local discriminator is employed to align local features, further enhancing domain invariance.
  • Point-wise Instance and Centroid Alignment (Novelty): This is the core contribution. PICA extends the point-wise alignment approach (like d-SNE) from classification to object detection.
  • Extra Classification Feature Layer: To better capture instance-level semantics, an additional feature layer is added to the classification branch of the detector. This layer's features are used for alignment.
  • Moving Average Centroid: To combat label noise in background ROIs and the scarcity of target instances (especially for rare classes), PICA computes moving average centroids for each category. These centroids are then incorporated into the alignment process. Crucially, noisy background ROI features are removed, and only the centroids are used for alignment.
  • Instance Balancing: To address category imbalance, undersampling is applied to the source and target features.
  • Improved Instance Alignment Loss: PICA proposes a modified instance alignment loss that not only maximizes the distance between different classes but also explicitly minimizes the absolute inter-class distance, improving class separation.

Datasets

The paper evaluates PICA on four datasets:

  • Cityscapes: A large-scale dataset of real-world urban street scenes with detailed annotations.
  • Foggy Cityscapes: A synthetic dataset derived from Cityscapes, simulating foggy conditions.
  • SIM10K: A dataset containing 10K synthetic images with bounding box annotations for cars, motorbikes, and persons.
  • Udacity: An open-source dataset from Udacity's self-driving car project, featuring diverse illumination and environmental conditions.

The experiments are conducted across three scenarios: Cityscapes to Foggy Cityscapes (C→F), SIM10K to Cityscapes (S→C), and Udacity to Cityscapes (U→C), simulating different types of domain shifts.

Key Insights and Ideas

  • Label Noise in Background is a Major Hurdle: The paper highlights that in loose annotation settings, background ROIs are a significant source of label noise, which can severely degrade the performance of instance-level alignment methods.
  • Centroids as Robust Anchors: Moving average centroids prove to be a robust way to represent object categories, mitigating the impact of noisy background labels and providing stable anchors for alignment, especially when target instances are scarce.
  • Separating Classification and Localization Features: Aligning features from the classification branch separately from localization features is more effective, as classification features for the same category are more semantically similar than localization features.
  • Point-wise Alignment Extends Beyond Classification: The paper successfully demonstrates that point-wise alignment, a technique proven effective in classification, can be adapted and extended to the more complex task of object detection.

Key Results

PICA achieves state-of-the-art performance in the few-shot domain adaptation setting:

  • C→F Scenario: PICA outperforms the state-of-the-art FDA method FAFRCNN significantly. It also shows a 1.6 AP gain over SWDA in the FUDA setting, demonstrating its effectiveness in leveraging scarce and loose target annotations.
  • S→C and U→C Scenarios: PICA achieves around 2.0 AP gain over FAFRCNN and 1.9 AP gain over SWDA on S→C. These results are particularly important as these scenarios involve only one foreground category, making background ROI consideration crucial.
  • Ablation Studies: The paper shows that each component of PICA (instance balancing, extra feature layer, moving average centroids, and the improved instance alignment loss) contributes to the overall performance improvement. Removing these components leads to significant performance drops.
  • Qualitative Results: PICA demonstrates improved bounding box prediction accuracy and the ability to detect objects that are challenging for other methods.

Limitations and Future Work

  • Small Bounding Box Mis-detection: The authors acknowledge that their model tends to mis-detect objects with small bounding boxes.
  • Multi-scale Feature Extraction: They suggest that incorporating multi-scale feature extraction could address the issue of small object detection.

Summary updated

BibTeX
@inproceedings{zhong2022pica,
  author = {Chaoliang Zhong and Jie Wang and Cheng Feng and Ying Zhang and Jun Sun and Yasuto Yokota},
  title = {{PICA: Point-Wise Instance and Centroid Alignment Based Few-Shot Domain Adaptive Object Detection With Loose Annotations}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure