Model Uncertainty based Active Learning on Tabular Data using Boosted Trees

Sharath M. Shankaranarayana

TL;DR

This paper introduces novel methods for active learning on tabular data using boosted trees, focusing on leveraging "model uncertainty" to intelligently select data for labeling. They propose new uncertainty-based sampling strategies and cost-effective active learning (CEAL) techniques for both classification and regression tasks, demonstrating their effectiveness compared to traditional methods.

What Problem Does This Paper Address?

Acquiring labeled data for supervised machine learning is often expensive and time-consuming, especially when domain experts are needed for annotation. Active Learning (AL) aims to mitigate this by intelligently selecting the most informative unlabeled data points for labeling, thereby maximizing model performance with a limited labeling budget. While AL has seen significant research, particularly for deep learning models on image and text data, its application to tabular data using gradient boosting models (like XGBoost, LightGBM, CatBoost) has been less explored. Traditional AL methods for classification often rely on entropy-based uncertainty, but this doesn't directly measure model uncertainty. This paper bridges this gap by exploring how to effectively use model uncertainty in AL for boosted trees on tabular data, addressing both classification and regression tasks.

Methods

The core of this paper lies in adapting and proposing new methods for active learning using model uncertainty with boosted trees, specifically CatBoost.

  • Model Uncertainty for Boosted Trees: The authors leverage recent advancements in quantifying model uncertainty for gradient boosting models. They explore two main approaches:
  • Virtual Ensembles [27]: This method estimates uncertainty from a single trained GBDT model by treating "truncated" sub-models as an ensemble. This is applied to both classification and regression.
  • Staged Predictions: For classification, they use the standard deviation of predictions from individual trees (similar to Monte-Carlo dropout for neural networks). For regression, they use the standard deviation of predictions from individual trees.
  • IBUG [5]: For regression, they also incorporate the Instance-Based Uncertainty estimation for Gradient-boosted regression trees (IBUG) method, which uses k-nearest neighbors to estimate uncertainty.
  • Uncertainty-Based Sampling: These model uncertainty measures are used as acquisition functions to score unlabeled data points. Points with higher uncertainty are selected for labeling.
  • Cost-Effective Active Learning (CEAL):
    • Classification: They propose an improved CEAL by introducing a two-stage filtering process. First, samples with low entropy (high confidence) are selected. Then, these high-confidence samples are further filtered based on low model uncertainty. This aims to generate more reliable pseudo-labels for cost-effective training.
  • Regression: A novel CEAL framework is proposed for regression. High-confidence samples (based on low model uncertainty) are identified and pseudo-labeled by the regressor. These pseudo-labeled samples are then added to the training set, reducing the need for expensive oracle annotations.

Datasets

The paper evaluates their proposed methods on several standard tabular datasets:

  • Classification:
    • Adult dataset (binary classification)
    • Amazon dataset (binary classification)
    • Thyroid dataset (multiclass classification)
  • Regression:
    • California Housing dataset
    • Energy dataset
    • Communities and Crime dataset

The datasets were split into 80% training and 20% testing. For active learning experiments, an initial training set of 20% of the training data was used, with subsequent data added incrementally.

Key Insights and Ideas

  • Model Uncertainty is a Viable Alternative to Entropy: The research demonstrates that model uncertainty, particularly when derived from virtual ensembles, can serve as an effective drop-in replacement for entropy-based uncertainty in active learning for boosted trees. This is significant because entropy doesn't directly measure model uncertainty.
  • CEAL Enhances Efficiency for Tabular Data: The proposed CEAL variants, especially those incorporating model uncertainty, show clear benefits in reducing the need for oracle labels while maintaining or improving model performance on tabular data.
  • Two-Stage Filtering for Reliable Pseudo-Labels: The novel two-stage filtering approach for CEAL in classification (entropy followed by model uncertainty) leads to more accurate pseudo-labels, mitigating the risk of noisy data introduced by overconfident models.
  • IBUG Shows Promise for Regression: For regression tasks, the IBUG method, combined with CEAL, demonstrates strong performance, converging to lower MSE scores.
  • Virtual Ensembles Offer a Good Trade-off: Virtual ensembles provide a computationally efficient way to estimate model uncertainty for boosted trees, offering a good balance between performance and complexity.

Key Results

  • Classification: Model uncertainty sampling strategies (especially those using virtual ensembles) generally perform comparably to or better than entropy-based sampling, and often outperform random sampling, particularly in low-data regimes. The proposed CEAL variants, especially the hybrid approach, show improved performance over traditional CEAL.
  • Regression: Model uncertainty sampling methods, particularly IBUG and virtual ensembles, show competitive results against greedy sampling and random sampling. The addition of pseudo-labels via CEAL consistently improves performance or matches the non-CEAL counterparts, with CEAL using virtual ensembles being particularly effective, especially with limited initial data.

Limitations and Future Work

  • Hyperparameter Tuning for CEAL: The authors acknowledge that CEAL methods, particularly the hybrid variants, require tuning of threshold parameters for high-confidence data selection. Reducing this hyperparameter dependency is a future direction.
  • Exploring Other Boosted Tree Methods: The work focuses on CatBoost. Future research could explore model uncertainty and CEAL with other recent boosted tree methods like NGBoost [9].
  • Passive vs. Active Sampling: While active sampling methods generally perform well, the paper notes that in some cases (e.g., Amazon dataset for classification), random sampling can be competitive or even better, highlighting the ongoing challenge of outperforming passive strategies in all scenarios.

Summary updated

Abstract

Supervised machine learning relies on the availability of good labelled data for model training. Labelled data is acquired by human annotation, which is a cumbersome and costly process, often requiring subject matter experts. Active learning is a sub-field of machine learning which helps in obtaining the labelled data efficiently by selecting the most valuable data instances for model training and querying the labels only for those instances from the human annotator. Recently, a lot of research has been done in the field of active learning, especially for deep neural network based models. Although deep learning shines when dealing with image\textual\multimodal data, gradient boosting methods still tend to achieve much better results on tabular data. In this work, we explore active learning for tabular data using boosted trees. Uncertainty based sampling in active learning is the most commonly used querying strategy, wherein the labels of those instances are sequentially queried for which the current model prediction is maximally uncertain. Entropy is often the choice for measuring uncertainty. However, entropy is not exactly a measure of model uncertainty. Although there has been a lot of work in deep learning for measuring model uncertainty and employing it in active learning, it is yet to be explored for non-neural network models. To this end, we explore the effectiveness of boosted trees based model uncertainty methods in active learning. Leveraging this model uncertainty, we propose an uncertainty based sampling in active learning for regression tasks on tabular data. Additionally, we also propose a novel cost-effective active learning method for regression tasks along with an improved cost-effective active learning method for classification tasks.
BibTeX
@inproceedings{shankaranarayana2023model,
  author = {Sharath M. Shankaranarayana},
  title = {{Model Uncertainty based Active Learning on Tabular Data using Boosted Trees}},
  booktitle = {Aimlsystems},
  year = {2023},
  doi = {10.1145/3639856.3639877},
}
Figure