Foreground Object Structure Transfer for Unsupervised Domain Adaptation

Jieren Cheng, Le Liu, Xiangyan Tang, Wenxuan Tu, Boyi Liu, Ke Zhou, Qiaobo Da, Yue Yang

Abstract

Unsupervised domain adaptation (UDA) aims to train a classification model from a labeled source domain for an unlabeled target domain. Since data distributions differ across domains, models often suffer from performance degradation. Existing methods typically align global feature distributions, which can lead to misalignment by ignoring class-specific information and background noise. We propose Foreground Object Structure Transfer (FOST), a method that leverages prior knowledge to extract foreground features and performs alignment within the local structure of objects. FOST utilizes a clustering-based condition combined with relative position relationships to make same-class data distributions more compact. Experimental results on benchmarks like ImageCLEF-DA and Office-Home demonstrate that FOST achieves state-of-the-art performance.

Introduction

Deep learning models often struggle with domain shift—the discrepancy between training (source) and deployment (target) data distributions. For instance, a classifier trained on studio photos may fail on sketches. Standard UDA techniques attempt to align these domains globally, but this approach faces two major challenges:

  1. Background Interference: Global alignment may inadvertently align irrelevant background features, leading to "negative transfer" where performance drops.
  2. Structural Information Loss: High-level semantic features often discard the spatial relationships (structure) between object parts, which are crucial for robust classification across domains.

Methodology: The FOST Framework

To address these issues, we introduce Foreground Object Structure Transfer (FOST). The framework focuses on aligning foreground object structures while suppressing background noise through three key innovations.

The FOST architecture integrates prior knowledge supervision and structural contrastive loss to align foreground object features.

Prior Knowledge Supervision

FOST identifies foreground objects by analyzing the gradients of ground-truth labels in the source domain. By weighting features based on their contribution to correct classification, the model focuses on "positive features" that represent the actual objects of interest, effectively creating a foreground mask without manual bounding boxes.

Structure Preservation

Unlike traditional methods that rely solely on the final, low-resolution feature maps of a backbone network, FOST merges features from earlier, shallower layers. These layers preserve the relative spatial positions of object parts (e.g., the eyes' position relative to the nose), providing a more stable structural representation for domain transfer than raw pixel values.

Structural Contrastive Loss (SCL)

In the absence of target labels, FOST employs spherical K-means clustering to assign pseudo-labels to target samples. The Structural Contrastive Loss (SCL) then minimizes intra-class distances and maximizes inter-class distances across both domains, ensuring a geometrically compact feature space.

Experimental Evaluation

We evaluated FOST on four standard UDA benchmarks: ImageCLEF-DA, Office-31, Office-Home, and VisDA-2017. FOST consistently outperformed state-of-the-art methods such as SRDC and CDAN, achieving an average accuracy of 95.3% on ImageCLEF-DA and 87.55% on the large-scale VisDA-2017 dataset.

Ablation Study

The ablation study highlights the importance of multi-layer feature integration. As shown in the results below, incorporating structural information from shallower layers ($S_2, S_3$) consistently improves the average accuracy compared to using only the deepest features ($S_4$).

Ablation study on the ImageCLEF-DA dataset using a ResNet-50 backbone, showing the performance gains from incorporating multi-layer structural features.

Conclusion

FOST introduces a novel approach to UDA by prioritizing foreground object structures and local geometric alignment over global statistical matching. By using gradients as a mask for prior knowledge and integrating shallow structural features, FOST achieves superior generalization across diverse datasets. Future work will focus on reducing the computational overhead for large-scale cross-domain transfers and improving the robustness of foreground extraction in low-data regimes.

Summary updated

Abstract

Unsupervised domain adaptation aims to train a classification model from the labeled source domain for the unlabeled target domain. Since the data distributions of the two domains are different, the model often performs poorly on the target domain. Existing methods align the feature distributions of the source and target domains and learn domain-invariant features to improve the performance of the model. However, the features are usually aligned as a whole, and the domain adaptation task fails to serve the classification, which will ignore the class information and lead to misalignment.In this paper, we investigate those features that should be used for domain alignment, introduce prior knowledge to extract foreground features to guide the domain adaptation task for classification tasks, and perform alignment in the local structure of objects. We propose a method called Foreground Object Structure Transfer(FOST). The key to FOST is the new clustering based condition, which combines the relative position relationship of foreground objects. Based on this conditions, FOST makes the data distribution of the same class more compact in geometry. In practice, since the label of the target domain is not available, we use the clustering information of the source domain to assign pseudo labels to the target domain samples, and then according to the source domain data prior knowledge guides those positive features to maximum the inter-class distance between different classes and mimimum the intra-class distance. Extensive experimental results on various benchmarks ($i.e.$ ImageCLEF-DA, Office-31, Office-Home, Visda-2017) under different domain adaptation settings prove that our FOST compares favorably against the existing state-of-the-art domain adaptation methods.
BibTeX
@misc{cheng2021foreground,
  author = {Jieren Cheng and Le Liu and Xiangyan Tang and Wenxuan Tu and Boyi Liu and Ke Zhou and Qiaobo Da and Yue Yang},
  title = {{Foreground Object Structure Transfer for Unsupervised Domain Adaptation}},
  year = {2021},
  eprint = {2109.06543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.06543},
}
Figure