Adversarial Sampling for Active Learning

Christoph Mayer, Radu Timofte

Abstract

This paper proposes ASAL, a new GAN based active learning method that generates high entropy samples. Instead of directly annotating the synthetic samples, ASAL searches similar samples from the pool and includes them for training. Hence, the quality of new samples is high and annotations are reliable. To the best of our knowledge, ASAL is the first GAN based AL method applicable to multi-class problems that outperforms random sample selection. Another benefit of ASAL is its small run-time complexity (sub-linear) compared to traditional uncertainty sampling (linear). We present a comprehensive set of experiments on multiple traditional data sets and show that ASAL outperforms similar methods and clearly exceeds the established baseline (random sampling). In the discussion section we analyze in which situations ASAL performs best and why it is sometimes hard to outperform random sample selection.

Introduction

Active Learning (AL) aims to train high-performing machine learning models using as little labeled data as possible by strategically choosing which samples an "oracle" (human annotator) should label. However, traditional AL methods face significant bottlenecks. Uncertainty sampling, for instance, requires scanning every image in an unlabeled pool to find informative samples, resulting in $O(n)$ linear complexity that fails at scale. Furthermore, previous Generative Adversarial Network (GAN) based AL approaches were often restricted to binary classification and required labeling potentially ambiguous synthetic images, leading to unreliable training signals.

Adversarial Sampling for Active Learning (ASAL) addresses these issues by using GANs to synthesize "uncertain" (high-entropy) data and matching these synthetic queries to real images in an unlabeled pool. This approach ensures high-quality training data while achieving sub-linear time complexity, making it uniquely suited for massive, multi-class datasets.

The ASAL Framework

ASAL introduces a three-step iterative cycle to overcome traditional hurdles:

  1. Adversarial Generation: A pre-trained GAN generates a synthetic image by optimizing a latent vector $z$ to maximize the entropy (uncertainty) of the current classifier. This identifies regions of the input space where the model is most confused.
  2. Sample Matching: To avoid the pitfalls of labeling "fake" images, ASAL identifies the most similar real images from the unlabeled pool. This matching uses feature extractors (e.g., autoencoders or the GAN's discriminator) and Principal Component Analysis (PCA) for efficient representation.
  3. Sub-linear Search: By employing k-d trees for the matching step, the search time is reduced to $O(\log n)$ relative to the pool size. This allows for near-instantaneous querying even in pools containing millions of images.
The ASAL framework: synthetic samples are generated to maximize classifier entropy and then matched to real samples in the pool.

Experimental Evaluation

The authors evaluated ASAL across several benchmarks, including standard vision sets (MNIST, CIFAR-10, SVHN) and large-scale datasets (CelebA, LSUN). Notably, LSUN provided a pool of approximately 10 million images to rigorously test runtime claims.

A critical finding was the importance of the feature space used for matching. While raw pixels suffice for simple datasets like MNIST, complex natural images in CelebA benefit significantly from features learned by an autoencoder or the GAN's own discriminator. This flexibility allows ASAL to adapt to various data distributions.

Key Results

ASAL demonstrated superior performance, outperforming random sampling in 8 out of 10 experimental benchmarks. For example, on a 2-class MNIST task, it reached 98.5% accuracy with only 200 samples, compared to the 500 required by random sampling.

Performance comparison on the MNIST 10-class dataset, showing ASAL variants consistently outperforming random sampling.

Beyond accuracy, ASAL offers a massive computational advantage. On the LSUN dataset with 16 million images, traditional uncertainty sampling took roughly 53 minutes to select 10 samples, whereas ASAL completed the task in just 44 seconds. This confirms that ASAL maintains sub-linear complexity, making its speed advantage grow even larger as the dataset size increases.

Discussion and Limitations

While ASAL represents a breakthrough in GAN-based AL, its success is tied to the underlying GAN's ability to accurately model the data distribution. If the GAN lacks diversity, the uncertainty search may become biased. Additionally, the method assumes a "dense" unlabeled pool where real images exist close to any synthetic high-entropy point. Future work may explore alternative AL scores, such as "Learning-Loss," and more advanced feature encoders to further enhance matching accuracy for complex datasets.

Summary updated

Abstract

This paper proposes ASAL, a new GAN based active learning method that generates high entropy samples. Instead of directly annotating the synthetic samples, ASAL searches similar samples from the pool and includes them for training. Hence, the quality of new samples is high and annotations are reliable. To the best of our knowledge, ASAL is the first GAN based AL method applicable to multi-class problems that outperforms random sample selection. Another benefit of ASAL is its small run-time complexity (sub-linear) compared to traditional uncertainty sampling (linear). We present a comprehensive set of experiments on multiple traditional data sets and show that ASAL outperforms similar methods and clearly exceeds the established baseline (random sampling). In the discussion section we analyze in which situations ASAL performs best and why it is sometimes hard to outperform random sample selection.
BibTeX
@inproceedings{mayer2020adversarial,
  author = {Christoph Mayer and Radu Timofte},
  title = {{Adversarial Sampling for Active Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=I3rjKJoFn_Ba},
}
Figure