stablelearner: Stability Assessment of Statistical Learning Methods

Michel Philipp, Carolin Strobl, Achim Zeileis, Thomas Rusch, Kurt Hornik, Lennart SchneiderOpenalex Unknown 2018 · DOI · OpenAlex

Abstract

Graphical and computational methods that can be used to assess the stability of results from supervised statistical learning.

Introduction

In machine learning, the primary focus is often on predictive accuracy. However, many powerful algorithms—particularly recursive partitioning methods like decision trees—can be "unstable." A small change in the training data, such as removing a few observations, can lead to a radically different model structure or set of predictions. This instability compromises interpretability; for instance, variable importance rankings might flip entirely with minor data tweaks. The stablelearner R package addresses this by providing standardized computational and graphical methods to measure and visualize this sensitivity, helping researchers determine if their model's insights are robust or merely artifacts of a specific training sample.

Methods

The package implements a general framework for stability assessment based on resampling. The core workflow involves:

  • Resampling: Generating multiple versions of the training data using techniques like bootstrapping, subsampling, jackknife, or split-half sampling.
  • Model Refitting: Automatically retraining the chosen "learner" (e.g., ctree, rpart, randomForest, svm, glm) on these resampled datasets.
  • Similarity Measurement: Quantifying the agreement between the results using measures like Cohen’s Kappa for classification or correlation coefficients for regression.

The resampling process is highly configurable, allowing users to specify the number of datasets and the nature of the sampling to be used in the assessment.

Resampling parameters in stablelearner

A specialized function, stabletree(), focuses on the stability of decision trees by tracking variable selection frequencies and cutpoint variability across resamples. The package is extensible, allowing users to register new learning algorithms via addLearner().

Datasets

The package includes several datasets for demonstration and benchmarking:

  • titanic: Survival data for 2,207 passengers and crew.
  • iris and airquality: Standard R datasets for classification and regression.
  • dgp_twoclass: A built-in data-generating function for creating artificial two-class problems with controllable noise and correlation.
Parameters for the dgp_twoclass data-generating function

Key Insights and Results

Stability is treated as a fundamental dimension of model quality. The package provides tools like plot.stabletree to visualize uncertainty, showing a "cloud" of cutpoints and variable selections rather than a single static tree. It also supports parallel evaluation via mclapply to handle the computational load of hundreds of resamples.

Quantitative stability profiles can be generated, including similarity matrices and variable selection frequencies (visualized through bar plots). Graphical analysis reveals if splits on specific predictors are consistent or fluctuate wildly across samples.

Limitations and Future Work

Current limitations include lack of support for interaction terms in certain coercions (e.g., ranger) and high computational requirements for very large datasets. While many standard learners are pre-registered, users must manually define prediction functions for newer or more exotic libraries. Future work aims to expand the library of supported learners and refine high-dimensional stability visualizations.

Summary updated

Abstract

Graphical and computational methods that can be used to assess the stability of results from supervised statistical learning.
BibTeX
@inproceedings{philipp2018stablelearner,
  author = {Michel Philipp and Carolin Strobl and Achim Zeileis and Thomas Rusch and Kurt Hornik and Lennart Schneider},
  title = {{stablelearner: Stability Assessment of Statistical Learning Methods}},
  booktitle = {Openalex Unknown},
  year = {2018},
  doi = {10.32614/cran.package.stablelearner},
}
Figure