Bayesian Generative Active Deep Learning

Toan Tran, Thanh-Toan Do, Ian D. Reid, Gustavo Carneiro

TL;DR

This paper introduces a novel framework that combines Active Learning and Data Augmentation to train deep neural networks using significantly less labeled data. By using a generative model (VAE-ACGAN) to produce synthetic samples specifically in the "uncertainty zones" of a classifier, the authors achieve higher accuracy with fewer human-labeled examples than traditional methods.

What Problem Does This Paper Address?

Deep learning is notoriously "data-hungry," requiring massive labeled datasets that are expensive and time-consuming to produce. Two common strategies to mitigate this are:

  1. Active Learning (AL): Having the model "ask" for labels on only the most informative samples from an unlabeled pool.
  2. Data Augmentation (DA): Artificially expanding the training set by creating new versions of existing data (e.g., flipping or rotating images).

However, both have flaws. Standard AL can still suffer from a lack of sufficient samples to define complex decision boundaries. Standard DA is often "blind"—it generates samples indiscriminately, many of which might be "easy" for the model and thus provide no new information. This paper addresses the gap: How can we generate new data that is guaranteed to be informative for the model?

Methods

The authors propose Bayesian Generative Active Deep Learning, which integrates three key components:

  • BALD (Bayesian Active Learning by Disagreement): An acquisition function that identifies "informative" samples—those where the model is most uncertain or where different versions of the model (sampled via Dropout) disagree the most.
  • VAE-ACGAN: A hybrid generative model combining a Variational Autoencoder (for reconstruction) and an Auxiliary Classifier GAN (for realistic generation).
Proposed Bayesian Generative Active Deep Learning workflow

The Workflow: 1. The system picks the most informative unlabeled sample ($x^*$) using BALD. 2. An "oracle" (human) labels it ($y^*$). 3. This sample is passed through the VAE-ACGAN to generate a synthetic "twin" ($x'$). 4. The model is then trained on both the real informative sample and its synthetic counterpart.

The core innovation is conditioning the generation on the most informative samples. Instead of generating random images, the model generates images specifically in the "disagreement region" where the classifier is struggling.

Datasets

The researchers evaluated their approach on four standard computer vision benchmarks:

  • MNIST: 60,000 handwritten digits.
  • CIFAR-10: 50,000 color images across 10 classes.
  • CIFAR-100: Similar to CIFAR-10 but with 100 finer-grained classes.
  • SVHN (Street View House Numbers): Over 73,000 images of house numbers from Google Street View.

Key Insights and Ideas

Illustration of generating synthetic samples within the disagreement region
  • Information-Preserving Augmentation: The authors mathematically prove (using Taylor expansion) that if a synthetic sample is generated in a small enough neighborhood of a highly informative real sample, the synthetic sample will also be highly informative. This ensures the generated data targets the classifier's weak spots.
  • Co-evolution: Unlike previous methods where the generator is pre-trained and static, this framework trains the classifier and the generator jointly. As the classifier gets smarter, the generator adapts to produce the specific types of "hard" samples the classifier currently finds confusing.
  • Efficiency over Volume: The results suggest that quality of data matters more than quantity. A few targeted synthetic samples are more effective for training than thousands of random augmentations.

Key Results

The proposed method (AL w. VAEACGAN) consistently outperformed standard Active Learning, random selection, and previous generative active learning approaches (like GAAL).

  • Accuracy Boost: On CIFAR-100 using a ResNet-18 architecture, the method achieved ~68% accuracy with only a fraction of the training data, significantly higher than the ~50% achieved by standard Active Learning without augmentation.
  • Convergence Speed: The model reaches competitive performance levels much faster. For instance, on MNIST, the active learning methods initially perform worse than random selection but quickly "cross over" and dominate once they have identified the core difficult samples (around 13% of the training set).
  • Superiority to BDA: It consistently beat Bayesian Data Augmentation (BDA), proving that conditioning generation on informative samples is better than just generating likely samples from the training distribution.

Limitations and Future Work

  • Computational Overhead: The authors acknowledge that their method is computationally more expensive than standard training because it requires running a VAE-ACGAN alongside the classifier.
  • Complexity of Acquisition: The current model relies on BALD. The authors suggest that future work could explore even more complex acquisition functions to guide the generation process.
  • Image Quality vs. Utility: While the generated images are "informative" for the classifier, they may not always look perfectly realistic to a human. The authors plan to investigate how to generate high-utility samples directly using more complex functions.

Summary updated

Abstract

Deep learning models have demonstrated outstanding performance in several problems, but their training process tends to require immense amounts of computational and human resources for training and...
BibTeX
@inproceedings{tran2019bayesian,
  author = {Toan Tran and Thanh-Toan Do and Ian D. Reid and Gustavo Carneiro},
  title = {{Bayesian Generative Active Deep Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=BkEuMibObB},
}
Figure