Abstract
While machine learning approaches to visual recognition offer great promise, most of the existing methods rely heavily on the availability of large quantities of labeled training data. However, in the vast majority of real-world settings, manually collecting such large labeled datasets is infeasible due to the cost of labeling data or the paucity of data in a given domain. In this paper, we present a novel Adversarial Knowledge Transfer (AKT) framework for transferring knowledge from internet-scale unlabeled data to improve the performance of a classifier on a given visual recognition task. The proposed adversarial learning framework aligns the feature space of the unlabeled source data with the labeled target data such that the target classifier can be used to predict pseudo labels on the source data. An important novel aspect of our method is that the unlabeled source data can be of different classes from those of the labeled target data, and there is no need to define a separate pretext task, unlike some existing approaches. Extensive experiments well demonstrate that models learned using our approach hold a lot of promise across a variety of visual recognition tasks on multiple standard datasets.
Introduction
Deep learning models are notoriously data-hungry, but manually labeling large datasets is expensive and time-consuming. To circumvent this, researchers often employ transfer learning, domain adaptation, or semi-supervised learning. However, these methods come with strict constraints: transfer learning requires massive labeled source datasets; domain adaptation assumes identical classes across domains; and semi-supervised learning requires the unlabeled data to share the same distribution as the labeled data.
We address a highly practical question: How can we efficiently transfer knowledge from random, "in-the-wild" unlabeled images to a specific target task without needing shared labels, shared distributions, or hand-crafted pretext tasks? Our proposed Adversarial Knowledge Transfer (AKT) framework leverages massive amounts of unlabeled data by aligning feature spaces to generate reliable pseudo-labels.
Methodology
The AKT framework is an end-to-end adversarial learning system that operates similarly to a Generative Adversarial Network (GAN). Instead of generating images, it generates pseudo-labels by aligning feature spaces between the source and target domains. The architecture consists of three primary modules:
- The Classifier ($M$): The primary network trained on the labeled target data.
- The Pseudo-Label Generator ($G$): A network mirroring $M$'s architecture that processes unlabeled source images to output features and pseudo-labels in the target domain's label space.
- The Discriminators ($D_I$ and $D_G$): Two networks designed to distinguish between features extracted from target data ($M$) and source data ($G$). $D_I$ operates at the instance level, while $D_G$ evaluates the mean features of a batch (group-level).
During training, the discriminators attempt to distinguish target features from source features, while the generator $G$ is trained to fool them. This adversarial process forces $G$ to extract features from the unlabeled source data that align with the distribution of the labeled target data. Once aligned, $G$ predicts pseudo-labels for the source data, which are then used alongside ground-truth target data to update the classifier $M$.
Experimental Results
We evaluated AKT across various visual recognition tasks, pairing labeled target datasets with different unlabeled source datasets, such as CIFAR-10 (Target) with CIFAR-100 (Source) and PASCAL-VOC (Target) with ImageNet (Source).
Key Insights
- Bypassing Pretext Tasks: AKT proves that complex, two-stage self-supervised pretext tasks (like image colorization) are not strictly necessary. Direct adversarial feature alignment offers a more streamlined training process.
- Cross-Semantic Transfer: Even when source data contains different objects, the network learns fundamental visual primitives (edges, textures, shapes) that boost target performance.
- Semantic Alignment: Because the generator is forced to assign target labels to source images, it makes visually logical associations. For instance, in PASCAL-VOC experiments, the generator often pseudo-labeled images of birds or dolphin fins as "aeroplanes" due to their visual similarity to wings.
Performance Metrics
AKT consistently outperformed self-supervised baselines. On CIFAR-10, AKT achieved 93.21% accuracy, surpassing methods like Split-Brain Autoencoders (92.60%). On the complex multi-label PASCAL-VOC dataset, AKT reached 77.4% mAP, significantly higher than the Rotation Decoupling method (74.5%). Ablation studies confirmed that removing adversarial feature alignment leads to a massive 10% performance drop on PASCAL-VOC, highlighting the necessity of alignment before pseudo-labeling out-of-distribution data.
Conclusion
Adversarial Knowledge Transfer provides a robust framework for leveraging unlabeled, "in-the-wild" data to improve specific visual recognition tasks. While a performance gap remains compared to fully supervised transfer learning, AKT demonstrates that visual primitives can be effectively transferred across semantic boundaries without the need for manual pretext task design. Future work may explore active filtering mechanisms to remove irrelevant source samples that could potentially degrade feature alignment.
Summary updated