Abstract
Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning (AL) provides a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle, maximizing mutual information such as BALD, has been successful and widely adapted. However, this pool-based objective inherently introduces redundant selection and requires high computational costs for batch selection. In this paper, we propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. By approximating each marginal distribution with a Beta distribution, we formulate BalEntAcq as a ratio between an augmented entropy and the marginalized joint entropy. BalEntAcq is a standalone measure that captures a well-diversified selection near the decision boundary with a margin. Experimental results on MNIST, CIFAR-100, SVHN, and TinyImageNet demonstrate that BalEntAcq consistently outperforms well-known linearly scalable AL methods.
Introduction
In modern machine learning, the bottleneck is often the high cost of data labeling. Active Learning (AL) addresses this by iteratively selecting the most informative samples from an unlabeled pool for human annotation. While Bayesian Active Learning by Disagreement (BALD) is a popular "info-max" approach, it faces two primary limitations: redundancy in batch selection and poor scalability due to complex relational computations. This work introduces BalEntAcq, a novel standalone metric that balances epistemic and aleatoric uncertainty to provide efficient, diverse data selection.
Methodology: Beta Distribution Approximation
The proposed method utilizes a Beta distribution approximation to model the marginal distributions of class probabilities in a Bayesian Neural Network (BNN). Using Monte Carlo (MC) Dropout, we obtain a set of softmax outputs for each input. Given that these probabilities are constrained to the interval [0, 1], the Beta distribution serves as an effective surrogate for the marginal distribution.
This approximation enables the derivation of a closed-form expression for Marginalized Joint Entropy (MJEnt). The resulting Balanced Entropy (BalEnt) ratio is defined as: $$BalEnt[x] := \frac{MJEnt[x]}{Ent[x] + \log 2}$$ This ratio identifies the equilibrium between model uncertainty and data noise. As a standalone measure, BalEntAcq avoids the $O(N^2)$ complexity of relational methods, allowing for linear scalability and parallel execution.
Key Insights: The Bifurcated Margin and Scalability
A distinctive feature of BalEntAcq is the "Bifurcated Margin." Unlike standard uncertainty measures that focus strictly on the decision boundary—often selecting noisy, unlearnable points—BalEntAcq selects points near the boundary with a slight margin. This behavior naturally promotes batch diversity without requiring explicit diversity constraints.
Furthermore, the computational efficiency of BalEntAcq is significant. In benchmarks on TinyImageNet, BalEntAcq processed the unlabeled pool in approximately 183 seconds, a fraction of the time required by methods like BADGE (4,936 seconds), while achieving comparable or superior performance.
Experimental Results
The efficacy of BalEntAcq was validated across several standard benchmarks:
- Accuracy Gains: On the MNIST dataset, BalEntAcq reached 96.5% accuracy with only 300 labeled points, surpassing BALD (95.3%) and Entropy (94.8%).
- Redundancy Resilience: In a "3×CIFAR-100" test (containing triplicate images), BalEntAcq maintained a high accuracy of 67.4%, while BALD's performance degraded to 64.6% due to the selection of redundant samples.
- Scalability: The method demonstrates linear scalability, making it suitable for large-scale production environments where batch selection speed is critical.
Conclusion
Balanced Entropy Acquisition (BalEntAcq) provides a robust and efficient framework for active learning in BNNs. By leveraging Beta distribution approximations, it achieves a balance between informativeness and diversity while maintaining linear computational complexity. Future work will investigate the application of this principle to non-vision domains and further refine the underlying mathematical interpretations of the information balance ratio.
Summary updated