Model-Agnostic Interpretability for Black-Box Systems

249 concepts201 papers1 proposal

As machine learning models grow in complexity—transitioning from simple linear regressions to deep neural networks and large language models—they often become "black boxes." While these systems achieve high accuracy, their internal decision-making logic is frequently opaque, creating a barrier to trust in high-stakes fields like healthcare, law, and autonomous systems.

Model-agnostic interpretability addresses this by treating the target model as a fixed oracle. Instead of inspecting internal weights or gradients, these methods analyze the relationship between input perturbations and output changes. This decoupling allows a single explanation framework to be applied across disparate architectures, from random forests to Transformers, without requiring access to proprietary internal parameters.

Probing the Decision Boundary

The core intuition behind model-agnostic explanation is that a model’s behavior can be characterized by how its predictions change in response to controlled interventions. Pedreschi et al. argue for a roadmap where transparency is reclaimed by interacting with models solely through inputs and outputs. This approach ensures that the explanation mechanism remains compatible with any architecture, effectively auditing the system from the outside.

One common technique is the use of local surrogates. As demonstrated by Heidari et al., one can explain complex predictions by fitting a simple, interpretable model (like a sparse linear regression) to the local neighborhood of a specific input. By observing how the black box responds to small perturbations around a data point, the surrogate captures the local decision logic without needing to understand the global complexity of the underlying model.

Counterfactuals and Contrastive Reasoning

A powerful way to explain a decision is to show what would have to change to reach a different outcome. This is known as counterfactual explanation, answering the human-centric question: "Why X rather than Y?"

Recent work has focused on making these "what-if" scenarios realistic and actionable:

  • Generative Approaches: Farid et al. and Madaan et al. utilize diffusion models to generate counterfactuals that respect the underlying data distribution, ensuring that the suggested changes are semantically meaningful rather than just adversarial noise.
  • Feasibility: Poyiadzi et al. introduced the FACE algorithm, which uses density-weighted graphs to find paths to a desired outcome that are supported by real data, ensuring the explanation is actionable in the real world.
  • Optimization: Sadiku proposed S-CFE, a framework that uses gradient-based optimization to find sparse counterfactuals—changing as few features as possible—while remaining model-agnostic.

These methods are particularly useful in specialized domains. For instance, Chu et al. applied counterfactual reasoning to graph neural networks for software vulnerability detection, identifying the minimal code changes needed to flip a "vulnerable" prediction to "non-vulnerable."

Axiomatic Foundations and Influence

To ensure that explanations are not just intuitive but also mathematically sound, researchers have turned to axiomatic approaches. By defining fundamental properties that an explanation must satisfy, they derive unique measures of feature influence.

Feng et al. established a mathematical foundation for concept explanations, interpreting decisions through human-friendly terms like "stripes" or "wings" by analyzing the statistical relationship between concept presence and model output. Similarly, Śliwiński et al. and Sliwinski et al. developed Monotone Influence Measures (MIM). These measures are derived from cooperative game theory and provide provably sound linear explanations based purely on the dataset structure, requiring no direct access to the model's internal logic.

Portability Across Domains

The primary advantage of the model-agnostic paradigm is its versatility. The same principles used to explain image classifiers can be adapted for:

  • Natural Language: Vafa et al. identify sequential rationales in LLMs by finding the smallest subsets of words that preserve a prediction, while Robeer et al. use GANs to generate fluent text perturbations.
  • Reinforcement Learning: Finkelstein et al. explain agent behavior by identifying environment modifications that align an agent's policy with human expectations.
  • Survival Analysis: Krzyziński et al. introduced SurvSHAP(t), a wrapper that provides time-dependent explanations for survival models by querying only the predicted survival function.

By decoupling the "explainer" from the "predictor," this research direction enables a unified approach to transparency. Whether auditing a proprietary API for bias using the Active Fourier Auditor Ajarra et al. or synthesizing realistic exemplars to explain skin lesion diagnosis Metta et al., model-agnostic interpretability ensures that as AI becomes more powerful, it does not become less understood.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_agnostic_explanation_e6a95656,
  title = {model-agnostic explanation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-agnostic-explanation/rdr_e6a9565651ca475e92f19b29f71eaaa1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_agnostic_explanation_e6a95656_v1,
  title = {model-agnostic explanation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-agnostic-explanation/rdr_e6a9565651ca475e92f19b29f71eaaa1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake