Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle

Jae Oh Woo

TL;DR

This paper introduces Balanced Entropy Acquisition (BalEntAcq), a novel uncertainty measure for Active Learning (AL) in Bayesian Neural Networks. By approximating softmax outputs with Beta distributions, the authors create a standalone, linearly scalable acquisition function that balances model uncertainty with label information, consistently outperforming traditional methods like BALD and Entropy across diverse image datasets.

What Problem Does This Paper Address?

Active Learning aims to maximize a model's performance while minimizing the number of expensive, human-labeled data points. A popular approach is the "info-max" principle (e.g., BALD), which selects points that maximize mutual information between model parameters and labels.

However, existing methods face two major hurdles:

  1. Redundancy and Scalability: Many high-performing methods (like BatchBALD or BADGE) require complex relational computations between data points, making them difficult to scale to large datasets or parallelize.
  2. The "Noise" Trap: Simple uncertainty measures often get stuck picking "noisy" data points (high aleatoric uncertainty) that don't actually help the model learn the underlying decision boundary.

The author, Jae Oh Woo, seeks a measure that is both computationally efficient (standalone) and "smart" enough to avoid redundant or non-informative samples.

Methods

The core innovation is the Balanced Entropy Learning Principle.

  • Beta Approximation: Softmax probabilities in Bayesian Neural Networks (BNNs) are typically hard to work with analytically. The author approximates these marginal distributions using Beta distributions. This allows for a closed-form expression of entropy.
  • Marginalized Joint Entropy (MJEnt): Using the Beta approximation, the paper derives MJEnt, which decomposes uncertainty into three parts: posterior (how much we expect to learn), epistemic (model uncertainty), and aleatoric (data noise).
  • BalEntAcq Function: The proposed acquisition function is a ratio between this marginalized joint entropy and an "augmented" entropy.
  • When the balance is positive, it takes the reciprocal to prioritize points where the "information imbalance" is greatest.
  • Crucially, it is a standalone measure: you can calculate the score for one image without looking at any others in the pool, making it perfectly suited for parallel processing.

Datasets

The authors evaluated the method on four standard computer vision benchmarks:

  • MNIST: The baseline for simple image classification.
  • CIFAR-100: A more complex dataset with 100 classes.
  • SVHN: Street View House Numbers, useful for testing robustness.
  • TinyImageNet: A subset of ImageNet with 200 classes, representing a more "real-world" scale.
  • 3×CIFAR-100: A custom "redundant" version of CIFAR-100 where images are tripled to test if the algorithm can avoid picking the same information multiple times.

Key Insights and Ideas

BalEntAcq selects points near the decision boundary with a bifurcated margin.
  • The Bifurcated Margin: Unlike BALD or Entropy, which tend to pick points directly on the decision boundary, BalEntAcq selects points near the boundary but with a specific margin. This "bifurcation" helps the model avoid points that are just noisy and instead focuses on points that define the shape of the boundary.
  • Standalone Diversity: Even though BalEntAcq doesn't explicitly calculate the distance between points (like BADGE does), it naturally produces a diversified selection. This is because it prioritizes a specific "contour" of information balance that is less likely to result in redundant clusters.
  • Efficiency is King: Because the formula is closed-form and standalone, it is significantly faster than non-linear methods. On TinyImageNet, BalEntAcq took about 183 seconds to select a batch, while the high-performing BADGE took nearly 4,936 seconds.

Key Results

  • Superior Accuracy: BalEntAcq consistently outperformed other linearly scalable methods (BALD, PowerBALD, MeanSD) across all datasets.
  • Redundancy Resilience: In the 3×CIFAR-100 experiment, BalEntAcq maintained high performance, proving it is less prone to selecting redundant information than standard BALD.
  • Competitive with Heavyweights: It performed on par with or better than BADGE (a state-of-the-art but computationally heavy method), especially as the training progressed. For example, on MNIST with only 300 labeled samples, it reached 96.5% accuracy.

Limitations and Future Work

  • BNN Dependency: The method relies on Bayesian Neural Networks (specifically using MC Dropout). While effective, this adds a computational overhead during the inference phase compared to standard "vanilla" neural networks.
  • Beta Approximation Limits: While the paper shows Beta distributions are a good fit for softmax outputs, this might not hold true for all types of architectures or non-image data.
  • Theoretical Open Questions: The author notes that the "negative probability" interpretation of the BalEnt ratio is still an open problem in information theory that requires further rigorous study.
  • Future Directions: The author suggests extending this principle to other Bayesian frameworks beyond simple classification, such as object detection or regression.

Summary updated

Abstract

Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle maximizing mutual information such as BALD has been successful and widely adapted in various active learning applications. However, this pool-based specific objective inherently introduces a redundant selection and further requires a high computational cost for batch selection. In this paper, we design and propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approximation enables us to formulate BalEntAcq as a ratio between an augmented entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAcq is a purely standalone measure without requiring any relational computations with other data points. Nevertheless, BalEntAcq captures a well-diversified selection near the decision boundary with a margin, unlike other existing uncertainty measures such as BALD, Entropy, or Mean Standard Deviation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.
BibTeX
@inproceedings{woo2023active,
  author = {Jae Oh Woo},
  title = {{Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ZTMuZ68B1g},
}
Figure