Active Learning with Model Selection

Alnur Ali, Rich Caruana, Ashish KapoorAAAI 2014 · OpenReview

Abstract

This chapter explores the integration of active learning and model selection, addressing the inherent dilemma between acquiring informative training data and maintaining unbiased validation sets. We present the Active Learning/Model Selection (ALMS) framework, which employs sequential and batch approaches to dynamically allocate a labeling budget. Through numerical examples, we demonstrate that ALMS effectively identifies optimal model architectures while outperforming traditional active learning methods.

Introduction

Active Learning (AL) is a powerful paradigm in machine learning designed to minimize labeling costs by selectively querying the most informative data points. However, a significant challenge often overlooked is the requirement to pre-specify a model architecture and its hyperparameters. If the chosen model is suboptimal for the task, the active learning process may be fundamentally flawed.

Standard active learning introduces selection bias; because queries focus on "difficult" or "uncertain" regions, the resulting training set is not representative of the underlying data distribution. This bias makes traditional cross-validation unreliable for model selection. Practitioners face a "Catch-22": using biased AL data for model selection leads to suboptimal choices, while diverting budget to a random validation set reduces the data available for training. The ALMS framework addresses this trade-off by treating model selection as an integral part of the active learning process.

Methodology: The ALMS Framework

The core of the ALMS algorithm is a Value of Information (VOI) approach that dynamically balances training and validation. ALMS maintains a candidate pool of diverse models (e.g., SVMs with varying kernels or Logistic Regression with different regularization parameters). At each iteration, the algorithm evaluates two competing objectives for every unlabeled point:

  1. Value of Training ($VOI_T$): The expected improvement in model accuracy if the point is added to the training set.
  2. Value of Validation ($VOI_V$): The expected utility of a random point in refining the estimate of which model is truly the best.

If $VOI_T > VOI_V$, ALMS queries an informative point for training; otherwise, it samples a random point for validation. To estimate model performance with minimal data, ALMS utilizes a nested Leave-2-Out Cross-Validation (L2OCV) technique.

The L2OCV procedure used in ALMS to estimate model performance and probabilities without bias from active sampling.

As shown in Figure 4, L2OCV allows the system to compute a distribution over models and estimate accuracies by rotating held-out points from the validation set. This ensures that model selection remains unbiased even as the training set becomes increasingly specialized.

Experimental Evaluation

The authors evaluated ALMS against several baselines, including Passive Learning (random sampling), Uncertainty Sampling, and Query by Committee (QBC). An "Oracle" baseline, which is pre-informed of the optimal hyperparameters, was also included. Testing was conducted on six binary classification datasets: USPS (digits), Ionosphere (radar), Mushroom (classification), Voting (records), Heart (disease), and Pima (diabetes).

Results and Discussion

ALMS consistently outperformed standard active learning baselines across all datasets. Notably, in several instances, ALMS matched or even exceeded the performance of the "Oracle." This suggests that the dynamic validation set helps better calibrate probability estimates than a fixed, pre-selected model.

Performance comparison across six datasets. Left column: Accuracy on test data vs. number of labels. Middle column: Allocation of budget between training and validation sets. Right column: Probability distribution over the model space.

Figure 5 illustrates the "Hedge Your Bets" strategy inherent in ALMS. Early in the labeling process, the algorithm allocates a larger portion of the budget to validation (red bars in the middle column) to resolve uncertainty about the best model. As confidence in a specific model architecture grows (right column), the budget shifts toward active training (blue bars). This behavior is dataset-specific; for example, the Mushroom dataset required more intensive training, while the Pima dataset demanded more careful validation for SVM calibration.

Limitations

While effective, ALMS has a computational cost associated with the L2OCV step, which scales quadratically with the size of the validation set ($O(|V_t|^2)$). Additionally, the VOI estimates can be noisy when very few labels are available, leading to initial instability in some scenarios.

Conclusion

The ALMS framework successfully bridges the gap between active learning and model selection. By dynamically splitting a single labeling budget, it avoids the pitfalls of biased validation while ensuring that the most informative data is used for training. Future work may explore importance weighting techniques to further optimize the use of actively sampled points for validation, potentially reducing the reliance on purely random sampling.

Summary updated

Abstract

This chapter contains sections titled: Direct Approach and the Active Learning/Model Selection Dilemma, Sequential Approach, Batch Approach, Ensemble Active Learning, Numerical Examples, Summary and Discussion
BibTeX
@inproceedings{ali2014active,
  author = {Alnur Ali and Rich Caruana and Ashish Kapoor},
  title = {{Active Learning with Model Selection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  url = {https://openreview.net/forum?id=SybLJeZd-H},
}
Figure