Machine learning often presents a frustrating trade-off: the models with the highest predictive accuracy—such as deep neural networks and complex ensembles—are typically the most difficult to understand. If a model's internal logic consists of millions of weights interacting in non-linear ways, it behaves as a "black box."
To resolve this tension, researchers have developed a paradigm known as post-hoc interpretability. Instead of constraining the model's architecture to be inherently simple (which often degrades performance), post-hoc methods allow you to train the most powerful model possible. Once training is complete, an external diagnostic layer is applied to explain the model's behavior without altering its underlying structure.
As surveyed by Madsen et al. and Longo et al., this approach decouples the training phase from the explanation phase. By treating the trained model as an environment to be probed, practitioners can maximize predictive performance in critical fields while still extracting human-centric insights.
Local Surrogates: Zooming In on a Single Decision
How do you explain a specific prediction from a model whose global decision boundary is too complex to write down? The core intuition behind local post-hoc interpretability is that complex boundaries look simple if you zoom in close enough.
While a deep neural network's global decision surface is highly non-linear, the boundary immediately surrounding a single data point can often be closely approximated by a simple linear model. This is the mechanism behind local surrogate frameworks like LIME.
Global Complex Boundary (Black Box) Local Linear Approximation (Surrogate)
_.._
/ \ / \ |
| \_________/ | ________|________
_______| |_______ / | \
|
To explain an individual prediction, a local surrogate framework systematically perturbs the input features around that specific instance and records how the black-box model's outputs change. As described by Mohamud et al. and Fellous et al., the framework then fits an inherently interpretable model (like a sparse linear regression) to these perturbed samples. The weights of this local model serve as feature importance scores, revealing exactly which inputs drove that specific decision—whether identifying household characteristics that signal economic vulnerability or pinpointing temporal windows in neural data. Because this process requires no access to internal model weights, it operates as a model-agnostic layer.
Global Proxies: Distilling the Network's Logic
When practitioners need to understand the overarching strategy of a model rather than a single prediction, they turn to global proxies or secondary models. The intuition here is akin to knowledge distillation, but optimized for human comprehension rather than computational efficiency.
Instead of forcing humans to interpret millions of raw parameters, a secondary, transparent model is trained to mimic the behavior of the primary black box. As demonstrated by Sagar et al. and Huang 0001 et al., researchers can feed data through a trained neural network, capture either its final outputs or its high-level latent features, and use them to train an inherently interpretable structure, such as a decision tree.
This secondary model acts as a simplified symbolic map. By translating the complex model's internal states into human-readable artifacts, visualizations, or linguistic labels, it delegates the final logical deduction to the user without degrading the primary system's operational efficiency (Doran et al.; Seong et al.).
Feature Attribution and Saliency Mapping
Another major post-hoc strategy focuses on feature attribution, which mathematically maps the final output back onto the input dimensions to highlight what the model "focused on."
Methods such as SHAP (SHapley Additive exPlanations) values and Partial Dependence Plots calculate the marginal contribution of each feature across various contexts to visualize the internal logic of complex algorithms (Zou). In domains like natural language processing or legal document analysis, this allows algorithms to isolate specific words, sentences, or higher-level concepts that heavily influence a classification (Mosca et al.; Prasad et al.). These techniques effectively generate a heatmap of importance, bridging the gap between high-dimensional computations and qualitative human reasoning.
The Catch: The Risk of Misalignment
While post-hoc interpretability provides a powerful window into black boxes, it introduces a fundamental risk: the explanation is an approximation, not the actual mechanism.
Because secondary models and local surrogates simplify the primary model's decision boundary to make it intelligible, they introduce a fidelity gap. As Vokinger et al. and Lima et al. caution, these post-hoc approximations carry the inherent danger of producing justifications that do not fully align with the primary model's true internal logic. If an explanation looks plausible but glosses over the model's actual, messy dependencies, it can create a false sense of security—potentially obscuring algorithmic bias or laundering developer accountability under the guise of transparency.
Key Foundational Work
Poverty Level Characterization via Feature Selection and Machine Learning (2019) — contributed 1 concept elements to this direction.
Explainable Artificial Intelligence for Neuroscience: Behavioral Neurostimulation (2019) — contributed 1 concept elements to this direction.
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives (2017) — contributed 1 concept elements to this direction.