Adversarial Knowledge Transfer from Unlabeled Data

Akash Gupta, Rameswar Panda, Sujoy Paul, Jianming Zhang, Amit K. Roy-Chowdhury

TL;DR

This paper introduces Adversarial Knowledge Transfer (AKT), a novel framework that improves a model's performance on a small labeled dataset by leveraging massive amounts of unlabeled data. Unlike existing methods, AKT does not require the unlabeled data to share the same classes or distribution as the target data, nor does it rely on manually designed "pretext" tasks, using adversarial feature alignment to generate reliable pseudo-labels instead.

What Problem Does This Paper Address?

Deep learning models are notoriously data-hungry, but manually labeling large datasets is expensive and time-consuming. To get around this, researchers use techniques like transfer learning, domain adaptation, and semi-supervised learning. However, these methods come with strict caveats:

  • Transfer Learning requires massive labeled source datasets (like ImageNet).
  • Domain Adaptation assumes the source and target datasets share the exact same classes.
  • Semi-Supervised Learning assumes the unlabeled data comes from the same distribution as the labeled data.
  • Self-Supervised Learning (using unlabeled data from anywhere) relies on "pretext tasks" (like solving jigsaw puzzles or colorizing images) to learn features. Designing these tasks is difficult, and training usually requires a clunky two-stage process (train on pretext, fine-tune on target).

The authors tackle a highly practical question: How can we efficiently transfer knowledge from random, "in-the-wild" unlabeled images to a specific target task, without needing shared labels, shared distributions, or hand-crafted pretext tasks?

Methods

The authors propose an end-to-end adversarial learning framework that operates like a GAN, but instead of generating images, it generates pseudo-labels by aligning feature spaces.

The architecture consists of three main modules:

  1. The Classifier ($M$): The main network we want to train, which processes the labeled target data.
  2. The Pseudo-Label Generator ($G$): A network with the same architecture as $M$. It takes unlabeled source images and outputs features and pseudo-labels in the target domain's label space.
  3. The Discriminators ($D_I$ and $D_G$): Two networks that try to tell the difference between features extracted by $M$ (target data) and features extracted by $G$ (source data). $D_I$ looks at individual instances, while $D_G$ looks at the mean features of a whole batch (group-level).

The Training Game: The discriminators are trained to distinguish target features from source features. Simultaneously, the generator $G$ is trained to fool the discriminators. This adversarial tug-of-war forces $G$ to extract features from the unlabeled source data that perfectly align with the distribution of the labeled target data. Once the features are aligned, $G$ can confidently predict pseudo-labels for the source data. Finally, the classifier $M$ is updated using both the ground-truth labeled target data and the pseudo-labeled source data.

Datasets

The authors tested their approach across various visual recognition tasks, pairing a labeled target dataset with a different, unlabeled source dataset:

  • Single-label Object Recognition: CIFAR-10 (Target) + CIFAR-100 (Source)
  • Multi-label Object Recognition: PASCAL-VOC (Target) + ImageNet (Source)
  • Font Character Recognition: Char74K (Target) + EMNIST (Source)
  • Handwritten Character Recognition: EMNIST (Target) + MNIST (Source)
  • Sentiment Recognition: Advertisement dataset (Target) + BAM dataset (Source)
  • Out-of-Distribution: EuroSAT (Target) + mini-ImageNet (Source)

Key Insights and Ideas

  • Pretext tasks aren't strictly necessary: The paper proves that you can bypass complex, two-stage self-supervised pretext tasks (like image colorization) by directly aligning feature spaces adversarially. This makes training much more streamlined.
  • Visual primitives transfer across semantic boundaries: Even if the source data contains completely different objects than the target data, the network still learns fundamental visual patterns (edges, textures, shapes) that boost target performance.
  • Semantic "bleeding" makes sense: Because the generator is forced to assign target labels to source images, it makes visually logical leaps. For example, when trained on PASCAL-VOC (which has an "aeroplane" class) and fed ImageNet data, the generator pseudo-labeled images of birds and dolphin fins as "aeroplanes" because of their visual similarity to wings.
  • Dual discriminators are better than one: Using both an instance-level and a group-level discriminator ensures that both localized, individual features and global, statistical distributions are aligned.

Key Results

  • Beats Self-Supervised Baselines: On CIFAR-10, AKT achieved 93.21% accuracy, outperforming state-of-the-art self-supervised methods like Split-Brain Autoencoders (92.60%) and Jigsaw (75.85%).
  • Strong on Complex Tasks: On the multi-label PASCAL-VOC dataset, AKT achieved 77.4% mAP, beating the competitive Rotation Decoupling method (74.5%).
  • Feature Alignment is Crucial: In ablation studies, removing the adversarial feature alignment caused a massive 10% drop in performance on PASCAL-VOC, proving that you cannot just blindly generate pseudo-labels for out-of-distribution data without aligning the feature spaces first.
  • Scales with Architecture: The method benefits from deeper networks. On CIFAR-10, moving from AlexNet to ResNet50 bumped accuracy from 89.04% to 94.65%.

Limitations and Future Work

While the paper does not feature a dedicated limitations section, a few constraints are apparent from the results. First, there is still a noticeable performance gap between this unsupervised transfer method and fully supervised transfer learning (e.g., 77.4% vs 87.0% on PASCAL-VOC), which is expected but highlights room for improvement on complex, multi-label tasks. Second, while the method doesn't require shared classes, it implicitly relies on the source and target domains sharing some underlying visual similarities (e.g., shapes or textures) for the pseudo-labeling to be beneficial rather than noisy. Future work could explore mechanisms to actively filter out completely irrelevant source samples that might degrade feature alignment.

Summary updated

Abstract

While machine learning approaches to visual recognition offer great promise, most of the existing methods rely heavily on the availability of large quantities of labeled training data. However, in the vast majority of real-world settings, manually collecting such large labeled datasets is infeasible due to the cost of labeling data or the paucity of data in a given domain. In this paper, we present a novel Adversarial Knowledge Transfer (AKT) framework for transferring knowledge from internet-scale unlabeled data to improve the performance of a classifier on a given visual recognition task. The proposed adversarial learning framework aligns the feature space of the unlabeled source data with the labeled target data such that the target classifier can be used to predict pseudo labels on the source data. An important novel aspect of our method is that the unlabeled source data can be of different classes from those of the labeled target data, and there is no need to define a separate pretext task, unlike some existing approaches. Extensive experiments well demonstrate that models learned using our approach hold a lot of promise across a variety of visual recognition tasks on multiple standard datasets.
BibTeX
@inproceedings{gupta2020adversarial,
  author = {Akash Gupta and Rameswar Panda and Sujoy Paul and Jianming Zhang and Amit K. Roy-Chowdhury},
  title = {{Adversarial Knowledge Transfer from Unlabeled Data}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413688},
}

Loading related content...

Figure