Towards Fair Knowledge Transfer for Imbalanced Domain Adaptation

Taotao Jing, Bingrong Xu, Zhengming Ding

TL;DR

The TFKT (Towards Fair Knowledge Transfer) framework addresses "transfer fairness" in domain adaptation where the source data is extremely imbalanced. By combining graph-based knowledge propagation, cross-domain feature augmentation, and a hybrid classifier system, it achieves over 20% accuracy improvements on standard benchmarks, even when some source categories have only a single labeled sample.

What Problem Does This Paper Address?

Domain Adaptation (DA) is a machine learning technique used when you have plenty of labeled data in one area (the source domain) but need to perform a task in a different, unlabeled area (the target domain). Most DA algorithms assume the source data is "ideal"—meaning it has a balanced number of examples for every category.

In the real world, data is often "long-tailed": a few categories have many samples, while most have very few. When a model is trained on this imbalanced source, it naturally favors the majority classes. When this biased knowledge is transferred to a target domain, the minority classes are essentially ignored, leading to a "fairness" issue where the model performs poorly on less-represented categories. The authors tackle this "extreme imbalance" scenario, specifically focusing on cases where minority classes might only have 1 or 5 labeled samples (1-shot or 5-shot learning).

Methods

The authors propose the Towards Fair Knowledge Transfer (TFKT) framework, which consists of three main pillars:

  1. Cross-Domain Feature Augmentation (CDA):
    • Knowledge Propagation: To deal with the noise and outliers common in few-shot samples, the model uses structure graphs (both within the source and across domains) to "smooth" the feature space. This helps minority samples move closer to their true class centers.
  2. Fulfillment Augmentation: The model generates "synthetic" features by interpolating (mixing) refined source embeddings with cross-domain information. This fills the distribution gap between the source and target domains.
  3. Hybrid Distinct Classifiers: Instead of one-size-fits-all, the framework uses two classifiers. A standard neural network classifier ($C_N$) handles the majority classes (where data is plentiful), while a prototypical classifier ($C_P$) handles the minority classes. Prototypical classifiers are more robust for few-shot learning because they classify based on distance to a "class center" rather than complex decision boundaries.
  4. Cross-Domain Prototype Alignment (CPA): The model aligns the source and target domains by minimizing the distance between their class prototypes (centers). This ensures that the "knowledge" of what a category looks like is consistent across both domains.

Datasets

The researchers evaluated TFKT on two popular computer vision benchmarks:

  • Office-31: Contains 4,110 images across 31 categories from three domains: Amazon, Webcam, and DSLR.
  • Office-Home: A more challenging dataset with 15,500 images across 65 categories from four domains: Artistic, Clip Art, Product, and Real-World.

To simulate extreme imbalance, they randomly selected only 1 or 5 samples for the first 10 (Office-31) or 25 (Office-Home) categories, treating the rest as the majority set.

Key Insights and Ideas

  • The "Fairness" Gap: Standard DA methods like DAN or SymNets actually perform quite poorly on minority classes because they are "distracted" by the majority distribution. TFKT shows that explicitly separating the treatment of majority and minority classes is essential.
  • Graph Smoothing as a Denoiser: In few-shot scenarios, a single "bad" sample can ruin a model. By using graph-based propagation, the model leverages the structure of the entire dataset (including unlabeled target data) to "correct" the position of those few labeled samples in the feature space.
  • Cross-Domain Mixup: Simply augmenting the source data isn't enough. By mixing source and target features during the augmentation phase, the model learns a "bridge" between domains that makes the final alignment much easier.

Key Results

TFKT significantly outperformed state-of-the-art models across the board:

  • Overall Accuracy: On the Office-Home 5-shot task, TFKT achieved 65.6% average accuracy, compared to 51.1% for DAN and 45.3% for SymNets.
  • Minority Class Performance: The most dramatic gains were in the minority categories ($A_f$). In some Office-Home tasks, TFKT improved minority accuracy by over 30% compared to traditional DA methods.
  • Robustness: Even in the "1-shot" scenario (only one labeled source sample per minority class), TFKT maintained stable performance, while other models' accuracies plummeted.

Limitations and Future Work

  • Computational Complexity: Building similarity graphs for very large datasets can be computationally expensive. While the authors use an "episodes" training strategy to mitigate this, scaling to millions of images would require further optimization.
  • Feature-Level vs. Image-Level: The augmentation happens at the "feature" level (vectors). While effective, it doesn't provide the interpretability of image-level generation (like GANs).
  • Future Directions: The authors suggest exploring more sophisticated generative strategies and applying this "fair transfer" logic to other tasks like object detection or semantic segmentation where class imbalance is also a major hurdle.

Summary updated

Abstract

Domain adaptation (DA) becomes an up-and-coming technique to address the insufficient or no annotation issue by exploiting external source knowledge. Existing DA algorithms mainly focus on practical knowledge transfer through domain alignment. Unfortunately, they ignore the fairness issue when the auxiliary source is extremely imbalanced across different categories, which results in severe under-presented knowledge adaptation of minority source set. To this end, we propose a Towards Fair Knowledge Transfer (TFKT) framework to handle the fairness challenge in imbalanced cross-domain learning. Specifically, a novel cross-domain knowledge propagation technique is proposed with the guidance of within-source and cross-domain structure graphs to smooth the manifold of the minority source set. Besides, a cross-domain fulfillment augmentation strategy is exploited achieve domain adaptation. Moreover, hybrid distinct classifiers and cross-domain prototype alignment are adopted to seek a more robust classifier boundary and mitigate the domain shift. Such three strategies are formulated into a unified framework to address the fairness issue and domain shift challenge. Extensive experiments over two popular benchmarks have verified the effectiveness of our proposed model by comparing to existing state-of-the-art DA models, and especially our model significantly improves over 20% on two benchmarks in terms of the overall accuracy.
BibTeX
@article{jing2021towards,
  author = {Taotao Jing and Bingrong Xu and Zhengming Ding},
  title = {{Towards Fair Knowledge Transfer for Imbalanced Domain Adaptation}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=YcWk_znxsG},
}

Loading related content...

Figure