Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle

Jae Oh Woo

Abstract

Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning (AL) provides a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle, maximizing mutual information such as BALD, has been successful and widely adapted. However, this pool-based objective inherently introduces redundant selection and requires high computational costs for batch selection. In this paper, we propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. By approximating each marginal distribution with a Beta distribution, we formulate BalEntAcq as a ratio between an augmented entropy and the marginalized joint entropy. BalEntAcq is a standalone measure that captures a well-diversified selection near the decision boundary with a margin. Experimental results on MNIST, CIFAR-100, SVHN, and TinyImageNet demonstrate that BalEntAcq consistently outperforms well-known linearly scalable AL methods.

Introduction

In modern machine learning, the bottleneck is often the high cost of data labeling. Active Learning (AL) addresses this by iteratively selecting the most informative samples from an unlabeled pool for human annotation. While Bayesian Active Learning by Disagreement (BALD) is a popular "info-max" approach, it faces two primary limitations: redundancy in batch selection and poor scalability due to complex relational computations. This work introduces BalEntAcq, a novel standalone metric that balances epistemic and aleatoric uncertainty to provide efficient, diverse data selection.

Methodology: Beta Distribution Approximation

The proposed method utilizes a Beta distribution approximation to model the marginal distributions of class probabilities in a Bayesian Neural Network (BNN). Using Monte Carlo (MC) Dropout, we obtain a set of softmax outputs for each input. Given that these probabilities are constrained to the interval [0, 1], the Beta distribution serves as an effective surrogate for the marginal distribution.

This approximation enables the derivation of a closed-form expression for Marginalized Joint Entropy (MJEnt). The resulting Balanced Entropy (BalEnt) ratio is defined as: $$BalEnt[x] := \frac{MJEnt[x]}{Ent[x] + \log 2}$$ This ratio identifies the equilibrium between model uncertainty and data noise. As a standalone measure, BalEntAcq avoids the $O(N^2)$ complexity of relational methods, allowing for linear scalability and parallel execution.

Correlation heatmap showing the relationship between Beta-approximated measures and standard uncertainty metrics.

Key Insights: The Bifurcated Margin and Scalability

A distinctive feature of BalEntAcq is the "Bifurcated Margin." Unlike standard uncertainty measures that focus strictly on the decision boundary—often selecting noisy, unlearnable points—BalEntAcq selects points near the boundary with a slight margin. This behavior naturally promotes batch diversity without requiring explicit diversity constraints.

Visualization of the selection strategy near decision boundaries.

Furthermore, the computational efficiency of BalEntAcq is significant. In benchmarks on TinyImageNet, BalEntAcq processed the unlabeled pool in approximately 183 seconds, a fraction of the time required by methods like BADGE (4,936 seconds), while achieving comparable or superior performance.

Experimental Results

The efficacy of BalEntAcq was validated across several standard benchmarks:

  • Accuracy Gains: On the MNIST dataset, BalEntAcq reached 96.5% accuracy with only 300 labeled points, surpassing BALD (95.3%) and Entropy (94.8%).
  • Redundancy Resilience: In a "3×CIFAR-100" test (containing triplicate images), BalEntAcq maintained a high accuracy of 67.4%, while BALD's performance degraded to 64.6% due to the selection of redundant samples.
  • Scalability: The method demonstrates linear scalability, making it suitable for large-scale production environments where batch selection speed is critical.

Conclusion

Balanced Entropy Acquisition (BalEntAcq) provides a robust and efficient framework for active learning in BNNs. By leveraging Beta distribution approximations, it achieves a balance between informativeness and diversity while maintaining linear computational complexity. Future work will investigate the application of this principle to non-vision domains and further refine the underlying mathematical interpretations of the information balance ratio.

Summary updated

Abstract

Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle maximizing mutual information such as BALD has been successful and widely adapted in various active learning applications. However, this pool-based specific objective inherently introduces a redundant selection and further requires a high computational cost for batch selection. In this paper, we design and propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approximation enables us to formulate BalEntAcq as a ratio between an augmented entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAcq is a purely standalone measure without requiring any relational computations with other data points. Nevertheless, BalEntAcq captures a well-diversified selection near the decision boundary with a margin, unlike other existing uncertainty measures such as BALD, Entropy, or Mean Standard Deviation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.
BibTeX
@inproceedings{woo2023active,
  author = {Jae Oh Woo},
  title = {{Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ZTMuZ68B1g},
}
Figure