Abstract
Graphical and computational methods that can be used to assess the stability of results from supervised statistical learning.
Introduction
In machine learning, the primary focus is often on predictive accuracy. However, many powerful algorithms—particularly recursive partitioning methods like decision trees—can be "unstable." A small change in the training data, such as removing a few observations, can lead to a radically different model structure or set of predictions. This instability compromises interpretability; for instance, variable importance rankings might flip entirely with minor data tweaks. The stablelearner R package addresses this by providing standardized computational and graphical methods to measure and visualize this sensitivity, helping researchers determine if their model's insights are robust or merely artifacts of a specific training sample.
Methods
The package implements a general framework for stability assessment based on resampling. The core workflow involves:
- Resampling: Generating multiple versions of the training data using techniques like bootstrapping, subsampling, jackknife, or split-half sampling.
- Model Refitting: Automatically retraining the chosen "learner" (e.g.,
ctree,rpart,randomForest,svm,glm) on these resampled datasets. - Similarity Measurement: Quantifying the agreement between the results using measures like Cohen’s Kappa for classification or correlation coefficients for regression.
The resampling process is highly configurable, allowing users to specify the number of datasets and the nature of the sampling to be used in the assessment.
A specialized function, stabletree(), focuses on the stability of decision trees by tracking variable selection frequencies and cutpoint variability across resamples. The package is extensible, allowing users to register new learning algorithms via addLearner().
Datasets
The package includes several datasets for demonstration and benchmarking:
titanic: Survival data for 2,207 passengers and crew.irisandairquality: Standard R datasets for classification and regression.dgp_twoclass: A built-in data-generating function for creating artificial two-class problems with controllable noise and correlation.
Key Insights and Results
Stability is treated as a fundamental dimension of model quality. The package provides tools like plot.stabletree to visualize uncertainty, showing a "cloud" of cutpoints and variable selections rather than a single static tree. It also supports parallel evaluation via mclapply to handle the computational load of hundreds of resamples.
Quantitative stability profiles can be generated, including similarity matrices and variable selection frequencies (visualized through bar plots). Graphical analysis reveals if splits on specific predictors are consistent or fluctuate wildly across samples.
Limitations and Future Work
Current limitations include lack of support for interaction terms in certain coercions (e.g., ranger) and high computational requirements for very large datasets. While many standard learners are pre-registered, users must manually define prediction functions for newer or more exotic libraries. Future work aims to expand the library of supported learners and refine high-dimensional stability visualizations.
Summary updated