TL;DR
This paper introduces the Mean Squared Shape function distance (MSS), a novel metric to quantify how much a model's internal structure changes during retraining. By focusing on "functionally transparent" models like Explainable Boosting Machines (EBMs), the authors demonstrate that simple strategies like sequential modeling can improve structural stability by up to 88% with negligible impact on predictive accuracy.
What Problem Does This Paper Address?
In the fast-paced world of general insurance, models are retrained frequently to account for new data and shifting trends. However, a common frustration for actuaries and domain experts is that a retrained model might produce similar overall results but arrive at them through a completely different internal logic. This is known as structural instability.
While "predictive stability" (how much the output changes) is well-studied, "structural stability" (how much the internal "shape" of the model changes) has been largely ignored outside of simple decision trees. If a model suddenly decides that a driver's age is less important than their car's color—despite no major change in the underlying data—it erodes trust and makes the model difficult to explain to regulators or customers. The authors aim to bridge this gap by providing a way to measure and improve the structural consistency of transparent machine learning models.
Methods
The authors first define Functional Transparency: a class of models where the prediction is a sum of univariate functions (main effects) and bivariate functions (pairwise interactions). This includes Generalised Linear Models (GLMs), Generalised Additive Models (GAMs), and Explainable Boosting Machines (EBMs).
Their core contribution is the Mean Squared Shape function distance (MSS). Unlike black-box metrics, MSS calculates the squared difference between the "shape functions" (the curves or steps that represent a feature's effect) of the pre-update and post-update models. This allows for stability analysis at three levels:
- Model-level: Overall stability of the system.
- Observation-level: Which specific policyholders see the most "logic shift"?
- Feature-level: Which specific variables (e.g., vehicle age) are causing the instability?
To test this, they developed a rigorous retraining framework to emulate real-world updates, as illustrated in their data-splitting methodology:
Datasets
The study utilizes three publicly available auto insurance datasets from the R package CASdatasets:
- norauto: 183,999 observations, 4 features.
- beMTPL97: 163,212 observations, 11 features.
- pg15training: 100,021 observations, 13 features.
For each dataset, the authors modeled three distinct insurance targets: claim frequency, claim severity, and total loss cost.
Key Insights and Ideas
- The Stability-Performance Trade-off: There is a clear hierarchy. GLMs are the most stable but often the least accurate. EBMs provide a "sweet spot," offering performance comparable to black-box models like XGBoost while remaining functionally transparent and structurally measurable.
- The Rashomon Effect: Multiple models can achieve near-identical predictive accuracy while having vastly different internal structures. MSS helps researchers choose the model that is not just accurate, but also "reliable" in its interpretation over time.
- Extrapolation is the Enemy: Feature-level analysis revealed that instability often occurs in the "tails" of data distributions (e.g., very old vehicles or rare geographic regions) where the model has fewer data points and is forced to extrapolate.
- Anchoring Works: "Sequential modeling"—where the new model is trained to predict the residuals of the old model (essentially anchoring it to the previous version)—is a powerful stabilizer.
Key Results
- Sequential Modeling: This strategy reduced the Mean Relative Dislocation (MRD) by at least 30% and up to 88% across various tests. Crucially, this massive gain in stability came with a deviance (error) increase of less than 1% in most cases.
- Fixed Hyperparameters: Simply reusing the hyperparameters from the previous model iteration (rather than re-tuning from scratch) significantly improved stability for EBMs without hurting performance.
- Model Ranking: In the benchmark on the
beMTPL97dataset, GLMs ranked #1 for stability, while EBMs outperformed GAMs in both stability and predictive power, effectively making GAMs obsolete for this use case.
Limitations and Future Work
- High-Order Interactions: The MSS metric is currently designed for models with up to pairwise interactions. It cannot yet quantify the structural stability of deep neural networks or gradient-boosted trees with high depth (e.g., XGBoost with depth > 2), as these involve complex, high-dimensional interactions that are difficult to visualize or measure as "shapes."
- Categorical Baseline Shifts: The authors noted that changes in "reference levels" for categorical variables can break the metric, requiring manual alignment of baselines before calculation.
- Future Directions: The authors suggest extending this framework to identify the "Rashomon set"—the collection of all models that are equally accurate—to specifically select the one with the most stable structure for long-term deployment.
Summary updated