Dropout-based Active Learning for Regression

Evgenii Tsymbalov, Maxim Panov, Alexander Shapeev

Abstract

Active learning is relevant and challenging for high-dimensional regression models when the annotation of the samples is expensive. Yet most of the existing sampling methods cannot be applied to large-scale problems, consuming too much time for data processing. In this paper, we propose a fast active learning algorithm for regression, tailored for neural network models. It is based on uncertainty estimation from stochastic dropout output of the network. Experiments on both synthetic and real-world datasets show comparable or better performance (depending on the accuracy metric) as compared to the baselines. This approach can be generalized to other deep learning architectures and offers a computationally efficient way of sampling additional data.

Introduction

In many machine learning domains, while raw data is abundant, obtaining high-quality labels is a significant bottleneck due to cost or time constraints. Active learning (AL) addresses this by intelligently selecting the most informative samples for labeling. While AL is well-established for classification, regression tasks—especially those involving high-dimensional data and large sample sizes—remain challenging. Traditional methods like Gaussian Processes or Bayesian Neural Networks provide robust uncertainty estimates but struggle to scale. This paper introduces MCDUE (Monte-Carlo Dropout Uncertainty Estimation), a lightweight algorithm designed to bridge the gap between computational efficiency and predictive performance in large-scale regression.

Methodology: MCDUE

The proposed MCDUE algorithm leverages Dropout, a standard regularization technique, to estimate model uncertainty during the inference phase.

  1. Training: A neural network is trained using dropout to prevent overfitting.
  2. Uncertainty Estimation: During inference on unlabeled data, dropout is kept active. Each sample is passed through the network $T$ times (e.g., $T=25$), generating a distribution of stochastic outputs.
  3. Acquisition: The standard deviation of these $T$ outputs serves as the uncertainty proxy. Samples with the highest standard deviation—indicating the model is most "uncertain"—are prioritized for labeling.

This method is highly efficient as it avoids the overhead of training ensembles or complex Bayesian architectures, requiring only multiple forward passes through a single trained model.

Experimental Setup

The authors evaluated MCDUE across seven diverse, high-dimensional datasets, primarily sourced from the UCI Machine Learning Repository. These benchmarks cover various domains, from medical imaging to social media analytics.

Table 1: Summary of the diverse real-world and synthetic datasets used in the study.

Key Results and Insights

The fundamental premise of MCDUE is that the variance in stochastic dropout outputs correlates with the actual prediction error. Empirical results confirm this strong relationship.

Figure 1: Correlation between MC Dropout standard deviation and absolute prediction error, demonstrating that higher variance is a reliable indicator of potential error.

Performance Benchmarks

  • Outperforming Randomness: MCDUE consistently surpassed random sampling across nearly all datasets. For instance, on the KEGG Network dataset, it achieved significantly lower error rates for the same labeling budget.
  • Efficiency and Scalability: Compared to "Greedy Max-Min" sampling, MCDUE provided competitive or superior accuracy (up to 20% improvement in some metrics) while being orders of magnitude faster to compute.
  • Metric Sensitivity: MCDUE is particularly effective at reducing MaxAE (Maximum Absolute Error), which is critical in fields like chemistry or physics where a single massive outlier can be catastrophic.
  • Robustness: Using Dolan-More curves to aggregate performance across all problems, MCDUE emerged as the most consistent top performer among the tested algorithms.

Conclusion and Future Work

MCDUE offers a practical, scalable solution for active learning in regression. By repurposing dropout for uncertainty estimation, it provides a "good enough" approximation that enables active learning on datasets where traditional Bayesian methods are computationally prohibitive. Future research will focus on the sensitivity of the method to hyperparameters like dropout probability and exploring automated ways to determine optimal batch sizes for data acquisition.

Summary updated

Abstract

Active learning is relevant and challenging for high-dimensional regression models when the annotation of the samples is expensive. Yet most of the existing sampling methods cannot be applied to large-scale problems, consuming too much time for data processing. In this paper, we propose a fast active learning algorithm for regression, tailored for neural network models. It is based on uncertainty estimation from stochastic dropout output of the network. Experiments on both synthetic and real-world datasets show comparable or better performance (depending on the accuracy metric) as compared to the baselines. This approach can be generalized to other deep learning architectures. It can be used to systematically improve a machine-learning model as it offers a computationally efficient way of sampling additional data.
BibTeX
@inproceedings{tsymbalov2018dropout,
  author = {Evgenii Tsymbalov and Maxim Panov and Alexander Shapeev},
  title = {{Dropout-based Active Learning for Regression}},
  booktitle = {AIST},
  year = {2018},
  url = {https://openreview.net/forum?id=HCd6jdXEcp},
}
Figure