Abstract
Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle maximizing mutual information such as BALD has been successful and widely adapted in various active learning applications. However, this pool-based specific objective inherently introduces a redundant selection and further requires a high computational cost for batch selection. In this paper, we design and propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approximation enables us to formulate BalEntAcq as a ratio between an augmented entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAcq is a purely standalone measure without requiring any relational computations with other data points. Nevertheless, BalEntAcq captures a well-diversified selection near the decision boundary with a margin, unlike other existing uncertainty measures such as BALD, Entropy, or Mean Standard Deviation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.
Introduction
Active learning (AL) is critical for improving data efficiency in scenarios where labeling is expensive. The goal is to identify a subset of unlabeled data that maximizes model performance when labeled. While the Bayesian Active Learning by Disagreement (BALD) objective is a standard choice, it faces significant practical hurdles: it often selects redundant points in batches and lacks computational scalability for large data pools.
To address these issues, this work introduces Balanced Entropy Acquisition (BalEntAcq). By utilizing a Beta distribution approximation, this method provides a standalone, parallelizable acquisition function that naturally ensures diversity and scales linearly with the size of the data pool.
Methodology: Beta Approximation and BalEntAcq
The core of the proposed approach is the Beta Approximation. The authors observe that the marginal distributions of softmax probabilities in Bayesian Neural Networks can be effectively modeled using Beta distributions. This allows for analytical, closed-form solutions for entropy calculations, bypassing the need for slow Monte Carlo sampling.
As shown in the correlation analysis above, the Beta-approximated measures (e.g., BetaMarginalBALD) exhibit a near-perfect correlation (0.99) with standard measures like BALD, justifying the use of this approximation for efficient computation.
The BalEntAcq measure is derived as a ratio between the Marginalized Joint Entropy (MJEnt) and an Augmented Shannon Entropy. By taking the reciprocal of this ratio, the acquisition function prioritizes points that balance model uncertainty with potential label information, avoiding regions of high aleatoric noise.
Key Insights: Diversity and Decision Boundaries
A unique property of BalEntAcq is its selection behavior near decision boundaries. While traditional measures like Entropy or BALD tend to pick points directly on the boundary—which are often the most noisy—BalEntAcq selects points in a "bifurcated" pattern, maintaining a slight margin from the boundary.
This geometric property allows BalEntAcq to produce diverse batches without requiring explicit comparisons between data points. This "standalone diversity" is a key factor in its superior performance and scalability compared to methods like BADGE or BatchBALD.
Experimental Evaluation
The method was rigorously tested on MNIST, CIFAR-100, SVHN, and TinyImageNet.
- Performance: BalEntAcq consistently outperformed BALD, MeanSD, and Entropy across all benchmarks.
- Redundancy Robustness: In a specialized "3x CIFAR-100" test with identical image copies, BalEntAcq maintained its efficiency, whereas BALD's performance dropped due to redundant selections.
- Computational Efficiency: On TinyImageNet, BalEntAcq demonstrated a 27x speedup over the non-linearly scalable BADGE method, while achieving higher accuracy.
Conclusion
BalEntAcq represents a significant step forward in Bayesian active learning, combining theoretical grounding with practical scalability. By leveraging the Beta approximation, it achieves state-of-the-art performance while remaining computationally efficient for large-scale datasets. Future research will explore extending this balanced entropy principle to other machine learning tasks beyond classification.
Summary updated