Interval Based Feature Importance Across Near Optimal Models

22 concepts11 papers1 proposal

When we train a machine learning model and ask, "Which features are most important?", standard interpretability tools provide a single list of scores. This point-estimate approach assumes a comfortable fiction: that there is one unique "best" model that captures the true relationship within the data.

In reality, real-world datasets often exhibit the Rashomon Effect—a phenomenon where many distinct models, utilizing entirely different predictive logic, achieve virtually identical accuracy. As argued by Rudin et al. in their position paper, Position: Amazing Things Come From Having Many Good Models, we should embrace this multiplicity rather than ignore it.

When multiple models perform equally well, traditional feature importance metrics become unstable and misleading. If two features are highly correlated, a standard sparse model might arbitrarily select one and assign it high importance while giving the other a score of zero. To resolve this ambiguity, a growing body of research shifts the paradigm from single-point importance scores to feature relevance intervals. By calculating the minimum and maximum possible importance a feature can take across all well-performing models, we can finally distinguish between features that are truly indispensable and those that are merely replaceable.

The Problem: Fragile Point Estimates

To understand why point estimates fail, consider a medical diagnosis dataset where two features—say, two different blood biomarkers—are highly correlated.

If we train a single decision tree or a lasso regression model, the algorithm will likely pick one biomarker to split on or assign a weight to, discarding the other to minimize model complexity. If we look only at this single "optimal" model, we would conclude that the chosen biomarker is critical and the discarded one is irrelevant.

However, as Xin et al. demonstrate in Exploring the Whole Rashomon Set of Sparse Decision Trees, there is often an entire set of near-optimal trees that swap these features out with no loss in predictive performance. Relying on a single model's feature importance makes our explanations fragile artifacts of the specific optimization seed or regularization penalty rather than true reflections of the underlying data structure.

The Solution: Model Class Reliance and Relevance Intervals

Instead of asking how much a specific trained model relies on a feature, we can ask: What is the range of reliance on this feature across all models that fit the data well?

This concept was formalized as Model Class Reliance (MCR) by Fisher et al. in All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance. Instead of a single score, MCR produces an interval:

$$[I_{\min}, I_{\max}]$$

Where $I_{\min}$ is the lowest possible importance assigned to a feature by any well-performing model in the class, and $I_{\max}$ is the highest. If a feature has an $I_{\min}$ near zero, it means the feature can be completely substituted by other variables without damaging the model's accuracy. This framework was later extended to non-linear ensembles like Random Forests by Smith et al. in Model Class Reliance for Random Forests, proving that these bounds can be computed efficiently even for complex, black-box model classes.

How It Works: Constrained Optimization

To compute these intervals without brute-force searching through an infinite space of models, the problem is framed as a constrained optimization task over the Rashomon set—the set of all models whose loss is within a small threshold $\epsilon$ of the empirical optimum.

   Model Parameter Space
  ┌───────────────────────────────┐
  │                               │
  │      Rashomon Set             │
  │     ┌──────────────┐          │
  │     │  f_min       │          │
  │     │  (Minimizes  │          │
  │     │   Importance)│          │
  │     │              │          │
  │     │        f_max │          │
  │     │   (Maximizes │          │
  │     │    Importance)          │
  │     └──────────────┘          │
  │                               │
  └───────────────────────────────┘

For linear classifiers, Göpfert et al. showed in Feature Relevance Bounds for Linear Classification that finding these bounds can be solved using efficient Linear Programming (LP). For each feature, the LP minimizes and maximizes that feature's weight, subject to the constraint that the overall model error does not exceed the acceptable threshold.

The authors extended this mathematical formulation to ordinal regression models in a series of papers (Pfannschmidt et al.; Pfannschmidt et al.), demonstrating that interval-based feature selection uncovers the complete set of informative features far more accurately than standard sparse regularization techniques like Lasso.

For non-linear models, the optimization requires different geometric insights. For instance, Zhong et al. in Exploring and Interacting with the Set of Good Sparse Generalized Additive Models approximate the Rashomon set of sparse Generalized Additive Models (GAMs) as a high-dimensional ellipsoid. This geometric approximation allows users to interactively compute the range of functional shapes and importances each variable can take, giving domain experts the power to audit and select models that align with human intuition.

A New Taxonomy of Feature Relevance

By looking at the specific boundaries of a feature's relevance interval, the authors of FRI-Feature Relevance Intervals for Interpretable and Interactive Data Exploration (Pfannschmidt et al.) established a rigorous three-tier taxonomy for features:

  1. Strongly Relevant (Essential): The lower bound is strictly greater than zero ($I_{\min} > 0$). This means every single model that performs well must rely on this feature. It cannot be substituted or ignored.
  2. Weakly Relevant (Redundant but Useful): The lower bound is zero, but the upper bound is greater than zero ($I_{\min} = 0, I_{\max} > 0$). There exist excellent models that completely ignore this feature, but there also exist excellent models that rely heavily on it. This is the hallmark of redundant or correlated features.
  3. Irrelevant: Both bounds are zero ($I_{\min} = 0, I_{\max} = 0$). No well-performing model utilizes this feature.
Strongly Relevant:  [=======] (Must be used)
Weakly Relevant:    [========] (Can be substituted)
Irrelevant:         []         (Never useful)
                    +--------+
                    0        Importance Scale

This taxonomy provides a much more honest foundation for domain-specific applications like biomarker discovery. Instead of arbitrarily picking one biomarker out of a correlated group, an interval-based approach flags the entire group as weakly relevant, preserving valuable biological alternatives for clinical consideration.

Quantifying Explanation Robustness

Interval-based thinking also serves as a powerful diagnostic tool for explanation reliability under environmental shifts.

Local explanation methods like SHAP scores are notoriously sensitive to the choice of the "background" data distribution used to simulate feature absence. Bertossi et al. address this vulnerability in The Distributional Uncertainty of the SHAP score in Explainable Machine Learning by calculating the minimum and maximum possible SHAP scores across a range of valid background distributions. The width of the resulting interval serves as a direct measure of explanation robustness: a narrow interval means the feature's importance is a stable property of the model, while a wide interval warns that the explanation is highly sensitive to arbitrary sampling assumptions.

Taking robustness a step further, Donnelly et al. introduced the UNIVERSE framework in Doctor Rashomon and the UNIVERSE of Madness: Variable Importance with Unobserved Confounding and the Rashomon Effect. This approach simultaneously bounds feature importance across both the Rashomon set of models and potential scenarios involving unobserved confounding (missing data). Under this rigorous formulation, a feature is only deemed truly essential if its lower bound remains strictly above zero across all valid models and all plausible confounding environments.

By shifting our perspective from fragile point estimates to robust relevance intervals, we move away from overconfident, arbitrary explanations and toward a deeper, more honest understanding of what our machine learning models are truly capable of learning from data.

Key Foundational Work

All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance (2018) — contributed 2 concept elements to this direction.
Feature Relevance Bounds for Linear Classification (2017) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_feature_relevance_intervals_bf72cf69,
  title = {feature relevance intervals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-relevance-intervals/rdr_bf72cf69609348c1bbe63f16b3314e52},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_feature_relevance_intervals_bf72cf69_v1,
  title = {feature relevance intervals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feature-relevance-intervals/rdr_bf72cf69609348c1bbe63f16b3314e52/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake