DAFD: Domain Adaptation via Feature Disentanglement for Image Classification

Zhize Wu, Changjiang Du, Le Zou, Ming Tan, Tongwen Xu, Fan Cheng, Fudong Nian, Thomas Weise

TL;DR

This paper introduces DAFD, a novel unsupervised domain adaptation method for image classification. DAFD disentangles category-relevant features from irrelevant ones (like backgrounds) using a coarse-to-fine approach, leading to improved accuracy on target domains with significant domain shifts.

What Problem Does This Paper Address?

Deep learning models for image classification often perform poorly when deployed in environments different from their training data. This "domain shift" is a major challenge because collecting and labeling large datasets for every possible scenario is impractical. Unsupervised Domain Adaptation (UDA) aims to bridge this gap by leveraging labeled data from a source domain to improve performance on an unlabeled target domain. Existing UDA methods often struggle because they try to align all features, including those irrelevant to classification (e.g., background clutter), which can interfere with learning domain-invariant representations. This paper addresses the problem of effectively aligning features in UDA by specifically focusing on disentangling and aligning only the most discriminative, category-relevant features.

Methods

The proposed method, Domain Adaptation via Feature Disentanglement (DAFD), employs a two-stage, coarse-to-fine strategy:

  1. Category-Relevant Feature Selection (CRFS) Module: This module is the "coarse-grained" component. It uses two feature encoders: one to extract category-relevant features (e.g., the object itself) and another for category-irrelevant features (e.g., background). By minimizing the mutual information between these two sets of features and using a reconstruction loss to ensure the irrelevant features are still meaningful, CRFS effectively "distills" the most important features for classification and discards noise. This prevents the network from being distracted by background or other non-discriminative elements.
  2. Dynamic Local Maximum Mean Discrepancy (DLMMD) Module: This is the "fine-grained" alignment component. After CRFS has isolated the relevant features, DLMMD further reduces the domain discrepancy within these features. Crucially, it performs this alignment at a category level. It uses pseudo-labels generated from the classifier's predictions on the target domain, but only for high-confidence predictions, to avoid negative impacts from misclassified samples. This ensures that features from the same category in different domains are brought closer together in the feature space.

The overall DAFD model is optimized by combining a classification loss, the DLMMD loss, a reconstruction loss, and a mutual information loss.

Datasets

The paper evaluates DAFD on four standard UDA benchmark datasets:

  • Office-31: A dataset for cross-domain image classification with 31 categories and three domains: Webcam (W), DSLR (D), and Amazon (A).
  • Office-Home: A more challenging dataset for UDA image classification with 65 categories and four domains: Clip Art (Cl), Artistic (Ar), Real-World (Rw), and Product (Pr).
  • MNIST: A dataset for digit recognition with 10 categories.
  • USPS: Another dataset for digit recognition with 10 categories.

The authors test all possible transfer tasks between domains within Office-31 and Office-Home, and also the MNIST↔USPS transfer tasks.

Key Insights and Ideas

  • Background is detrimental to UDA: The paper strongly argues that aligning background and other category-irrelevant features actively harms domain adaptation performance, as these features are inherently domain-specific and noisy.
  • Coarse-to-fine disentanglement is effective: Separating the task into a coarse feature selection stage (CRFS) and a fine-grained alignment stage (DLMMD) allows for more targeted and effective domain adaptation.
  • Category-level alignment is crucial: Aligning features only within the same category (as done by DLMMD) prevents the model from collapsing different categories into a single cluster, which is a common pitfall in standard discrepancy-based methods.
  • Mutual Information for Feature Distillation: Using mutual information to encourage disentanglement between relevant and irrelevant features is a novel way to guide the network towards learning more robust representations.

Key Results

  • DAFD consistently outperforms strong baselines like DSAN on Office-31 and Office-Home, showing significant improvements in average accuracy (e.g., 1.4% on Office-31, 1.9% on Office-Home).
  • The method achieves state-of-the-art or near state-of-the-art results across most tasks and datasets, demonstrating its robustness. For instance, on Office-31, the average accuracy reaches 89.8%, and on Office-Home, it's 69.5%.
  • Ablation studies confirm the effectiveness of both the CRFS and DLMMD modules. Removing DLMMD still shows improvement over baselines, but combining both modules yields the best results.
  • Visualization (CAM and t-SNE) clearly shows that CRFS focuses attention on discriminative foreground features and that DAFD effectively aligns source and target domain data points.

Limitations and Future Work

  • Pseudo-labeling threshold: The reliance on a fixed probability threshold (K=0.8) for pseudo-labeling in DLMMD might be a point of sensitivity.
  • Future Directions: The authors plan to explore other feature disentanglement techniques for even better results. They also intend to extend their method to other tasks like object detection and investigate combining DAFD with other advanced UDA methods like DALN.

Summary updated

Abstract

A good feature representation is the key to image classification. In practice, image classifiers may be applied in scenarios different from what they have been trained on. This so-called domain shift leads to a significant performance drop in image classification. Unsupervised domain adaptation (UDA) reduces the domain shift by transferring the knowledge learned from a labeled source domain to an unlabeled target domain. We perform feature disentanglement for UDA by distilling category-relevant features and excluding category-irrelevant features from the global feature maps. This disentanglement prevents the network from overfitting to category-irrelevant information and makes it focus on information useful for classification. This reduces the difficulty of domain alignment and improves the classification accuracy on the target domain. We propose a coarse-to-fine domain adaptation method called Domain Adaptation via Feature Disentanglement~(DAFD), which has two components: (1)the Category-Relevant Feature Selection (CRFS) module, which disentangles the category-relevant features from the category-irrelevant features, and (2)the Dynamic Local Maximum Mean Discrepancy (DLMMD) module, which achieves fine-grained alignment by reducing the discrepancy within the category-relevant features from different domains. Combined with the CRFS, the DLMMD module can align the category-relevant features properly. We conduct comprehensive experiment on four standard datasets. Our results clearly demonstrate the robustness and effectiveness of our approach in domain adaptive image classification tasks and its competitiveness to the state of the art.
BibTeX
@misc{wu2023dafd,
  author = {Zhize Wu and Changjiang Du and Le Zou and Ming Tan and Tongwen Xu and Fan Cheng and Fudong Nian and Thomas Weise},
  title = {{DAFD: Domain Adaptation via Feature Disentanglement for Image Classification}},
  year = {2023},
  eprint = {2301.13337},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.13337},
}
Figure