Dropout-Based Active Learning for Regression

Evgenii Tsymbalov, Maxim Panov, Alexander Shapeev

TL;DR

This paper introduces MCDUE (Monte-Carlo Dropout Uncertainty Estimation), a computationally efficient active learning algorithm specifically designed for high-dimensional regression tasks. By using dropout at the prediction stage to estimate model uncertainty, the authors provide a way to select the most informative data points for labeling, significantly outperforming random sampling and matching or beating more complex baselines with much lower overhead.

What Problem Does This Paper Address?

In many machine learning applications, data is plentiful, but labels are expensive. Active learning (AL) attempts to solve this by having the model "ask" for the labels of the most informative samples.

While AL is well-studied for classification, regression presents unique challenges. Existing methods like Gaussian Processes or Bayesian Neural Networks provide excellent uncertainty estimates but become computationally intractable as the number of samples ($>10^4$) or input dimensions ($>10$) grows. Ensemble methods (training multiple models) are another alternative, but they are time-consuming to train. This paper seeks a "middle ground": an algorithm that is fast enough for large-scale deep learning but accurate enough to significantly reduce the amount of labeled data needed.

Methods

The authors propose MCDUE, which leverages a common regularization technique called Dropout in a novel way for regression.

  1. Standard Training: A neural network is trained with dropout (randomly "muting" neurons) to prevent overfitting.
  2. Uncertainty Estimation (The "MC" part): Instead of turning dropout off during inference (the standard approach), the authors keep it on. They run the same unlabeled sample through the network $T$ times (e.g., 25 times). Because different neurons are muted each time, the model produces slightly different outputs.
  3. Acquisition Function: The algorithm calculates the standard deviation of these $T$ stochastic outputs. A high standard deviation suggests the model is "confused" or uncertain about that specific data point.
  4. Sampling: The points with the highest standard deviation are selected from the unlabeled "pool" to be annotated by an expert and added to the training set.

This approach is "lightweight" because it doesn't require training multiple independent models; it only requires multiple forward passes through a single trained model.

Datasets

The authors evaluated their approach on seven diverse datasets, primarily from the UCI Machine Learning Repository, representing real-world high-dimensional problems:

  • BlogFeedback: Predicting the number of comments on blog posts.
  • SGEMM GPU: Measuring GPU performance/calculation time.
  • YearPredictionMSD: Predicting the release year of songs.
  • Relative location of CT slices: Spatial data from medical imaging.
  • Online News Popularity: Predicting social media shares.
  • KEGG Network: Biochemical pathway data.
  • Rosenbrock 2000D: A high-dimensional (2000 attributes) synthetic benchmark.

Key Insights and Ideas

  • The Uncertainty-Error Correlation: The core "aha" moment is the empirical observation that dropout-based variance correlates strongly with actual prediction error in regression. If the model's "stochastic guesses" vary wildly, it’s a very good indicator that the model's average prediction is likely wrong.
  • Efficiency over Complexity: While Bayesian Neural Networks are theoretically more "correct" for uncertainty, MCDUE provides a "good enough" approximation that scales to hundreds of thousands of samples where Bayesian methods would fail.
  • Metric Sensitivity: The authors highlight that the choice of metric matters. MCDUE is particularly effective at reducing MaxAE (Maximum Absolute Error), which is critical in fields like chemistry or physics where a single massive outlier can be catastrophic.

Key Results

  • Superiority over Randomness: In almost all tests, MCDUE outperformed random sampling significantly. For example, on the KEGG Network dataset, the error ratio showed MCDUE was substantially more accurate for the same number of labeled samples.
  • Competitive with Max-Min: MCDUE was compared against "Greedy Max-Min" sampling (which picks points furthest away from the current training set in the input space). MCDUE slightly outperformed Max-Min in accuracy (up to 20% better in some metrics) while being significantly faster to compute.
  • Dolan-More Analysis: Using Dolan-More curves (a way to compare algorithms across many problems), the authors showed that MCDUE was the "best" performer (the most frequent winner) across their entire suite of datasets.

Limitations and Future Work

  • Hyperparameter Sensitivity: The effectiveness of the method depends on the dropout probability ($\pi$) and the network architecture. The authors note that these require further systematic study.
  • "Ad Hoc" Nature: The paper acknowledges that this is an empirical, ad hoc approach. While it works well in practice, the theoretical link between dropout variance and true model uncertainty in a regression context needs deeper investigation.
  • Batch Size: Future work could involve automatically estimating how many samples to acquire in each step, rather than using a fixed number.

Summary updated

Abstract

Active learning is relevant and challenging for high-dimensional regression models when the annotation of the samples is expensive. Yet most of the existing sampling methods cannot be applied to large-scale problems, consuming too much time for data processing. In this paper, we propose a fast active learning algorithm for regression, tailored for neural network models. It is based on uncertainty estimation from stochastic dropout output of the network. Experiments on both synthetic and real-world datasets show comparable or better performance (depending on the accuracy metric) as compared to the baselines. This approach can be generalized to other deep learning architectures. It can be used to systematically improve a machine-learning model as it offers a computationally efficient way of sampling additional data.
BibTeX
@inproceedings{tsymbalov2018dropout,
  author = {Evgenii Tsymbalov and Maxim Panov and Alexander Shapeev},
  title = {{Dropout-Based Active Learning for Regression}},
  booktitle = {AIST},
  year = {2018},
  url = {https://openreview.net/forum?id=HCd6jdXEcp},
}

Loading related content...

Figure