Quantifying structural stability of transparent models for general insurance pricing

Félix Côté, Marie-Pier CôtéarXiv 2026 · arXiv

TL;DR

This paper introduces the Mean Squared Shape function distance (MSS), a novel metric to quantify how much a model's internal structure changes during retraining. By focusing on "functionally transparent" models like Explainable Boosting Machines (EBMs), the authors demonstrate that simple strategies like sequential modeling can improve structural stability by up to 88% with negligible impact on predictive accuracy.

What Problem Does This Paper Address?

In the fast-paced world of general insurance, models are retrained frequently to account for new data and shifting trends. However, a common frustration for actuaries and domain experts is that a retrained model might produce similar overall results but arrive at them through a completely different internal logic. This is known as structural instability.

While "predictive stability" (how much the output changes) is well-studied, "structural stability" (how much the internal "shape" of the model changes) has been largely ignored outside of simple decision trees. If a model suddenly decides that a driver's age is less important than their car's color—despite no major change in the underlying data—it erodes trust and makes the model difficult to explain to regulators or customers. The authors aim to bridge this gap by providing a way to measure and improve the structural consistency of transparent machine learning models.

Methods

The authors first define Functional Transparency: a class of models where the prediction is a sum of univariate functions (main effects) and bivariate functions (pairwise interactions). This includes Generalised Linear Models (GLMs), Generalised Additive Models (GAMs), and Explainable Boosting Machines (EBMs).

Their core contribution is the Mean Squared Shape function distance (MSS). Unlike black-box metrics, MSS calculates the squared difference between the "shape functions" (the curves or steps that represent a feature's effect) of the pre-update and post-update models. This allows for stability analysis at three levels:

  1. Model-level: Overall stability of the system.
  2. Observation-level: Which specific policyholders see the most "logic shift"?
  3. Feature-level: Which specific variables (e.g., vehicle age) are causing the instability?

To test this, they developed a rigorous retraining framework to emulate real-world updates, as illustrated in their data-splitting methodology:

Figure 1: Construction of the pre-update and post-update training and test datasets using a time stamp.

Datasets

The study utilizes three publicly available auto insurance datasets from the R package CASdatasets:

  • norauto: 183,999 observations, 4 features.
  • beMTPL97: 163,212 observations, 11 features.
  • pg15training: 100,021 observations, 13 features.

For each dataset, the authors modeled three distinct insurance targets: claim frequency, claim severity, and total loss cost.

Key Insights and Ideas

  • The Stability-Performance Trade-off: There is a clear hierarchy. GLMs are the most stable but often the least accurate. EBMs provide a "sweet spot," offering performance comparable to black-box models like XGBoost while remaining functionally transparent and structurally measurable.
  • The Rashomon Effect: Multiple models can achieve near-identical predictive accuracy while having vastly different internal structures. MSS helps researchers choose the model that is not just accurate, but also "reliable" in its interpretation over time.
  • Extrapolation is the Enemy: Feature-level analysis revealed that instability often occurs in the "tails" of data distributions (e.g., very old vehicles or rare geographic regions) where the model has fewer data points and is forced to extrapolate.
  • Anchoring Works: "Sequential modeling"—where the new model is trained to predict the residuals of the old model (essentially anchoring it to the previous version)—is a powerful stabilizer.

Key Results

  • Sequential Modeling: This strategy reduced the Mean Relative Dislocation (MRD) by at least 30% and up to 88% across various tests. Crucially, this massive gain in stability came with a deviance (error) increase of less than 1% in most cases.
  • Fixed Hyperparameters: Simply reusing the hyperparameters from the previous model iteration (rather than re-tuning from scratch) significantly improved stability for EBMs without hurting performance.
  • Model Ranking: In the benchmark on the beMTPL97 dataset, GLMs ranked #1 for stability, while EBMs outperformed GAMs in both stability and predictive power, effectively making GAMs obsolete for this use case.

Limitations and Future Work

  • High-Order Interactions: The MSS metric is currently designed for models with up to pairwise interactions. It cannot yet quantify the structural stability of deep neural networks or gradient-boosted trees with high depth (e.g., XGBoost with depth > 2), as these involve complex, high-dimensional interactions that are difficult to visualize or measure as "shapes."
  • Categorical Baseline Shifts: The authors noted that changes in "reference levels" for categorical variables can break the metric, requiring manual alignment of baselines before calculation.
  • Future Directions: The authors suggest extending this framework to identify the "Rashomon set"—the collection of all models that are equally accurate—to specifically select the one with the most stable structure for long-term deployment.

Summary updated

Abstract

The short duration of general insurance contracts leads to rapid distributional shifts in the data, warranting regular model updates. This process may yield a post-update model that differs considerably from the pre-update model in both predictions and structure. Structural stability is upstream of predictive stability, yet existing structural stability metrics are mostly restricted to simple decision trees. Thus, we first introduce the notion of functional transparency, characterising models with a prediction function that can be expressed as a sum of univariate functions for main effects and bivariate functions for pairwise interactions. Then, we propose the Mean Squared Shape function distance, a metric to quantify the structural stability of functionally transparent models, which are particularly relevant in actuarial science. Using three public auto insurance datasets, we present a benchmark study to demonstrate the usefulness of our metric in identifying a stable and accurate transparent model. Generalised linear models with centred numerical covariates are stable, and explainable boosting machines (EBMs) offer a compromise between stability and performance. We also find that stabilisation strategies, such as sequential modelling and fixed hyperparameters, can substantially improve stability for both EBM and XGBoost models, with negligible impact on performance.
BibTeX
@misc{cote2026quantifying,
  author = {Félix Côté and Marie-Pier Côté},
  title = {{Quantifying structural stability of transparent models for general insurance pricing}},
  year = {2026},
  eprint = {2611.17308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17308},
}
Figure