Machine learning has long operated under an implicit assumption: to achieve the highest predictive accuracy, you must accept a "black box." High-capacity models like deep neural networks or gradient-boosted tree ensembles learn intricate, high-dimensional representations that humans cannot easily trace. This opacity creates severe risks in high-stakes domains like healthcare, finance, and autonomous driving, where understanding why a decision was made is legally or operationally mandatory (Buczynski et al.; Singh et al.).
However, a growing body of research challenges this false dichotomy. Instead of forcing a choice between a highly accurate black box and a simplistic, low-capacity "glass box" (like a linear regression), modern architectures achieve state-of-the-art performance while remaining inherently interpretable. They do this by structurally constraining how information flows through the model.
Isolating Features via Neural Additive Structures
The primary culprit behind model opacity is the uncontrolled explosion of high-order feature interactions inside hidden layers. If features $x_1$ and $x_2$ are mixed non-linearly across dozens of layers, it becomes impossible to isolate the impact of $x_1$ alone on the final output.
To solve this, Lou et al. revitalized Generalized Additive Models (GAMs). A GAM expresses the target variable as a sum of independent univariate functions:
$$f(x) = f_1(x_1) + f_2(x_2) + \dots + f_d(x_d)$$
Because there are no feature interactions, a human can plot each "shape function" $f_i$ independently to see exactly how that feature influences the prediction. Lou et al. demonstrated that by using gradient-boosted trees to learn these shape functions, GAMs could match the accuracy of complex random forests on tabular data while remaining fully transparent.
Input Features Shape Functions (Neural Nets) Output
[ x_1 ] ---------> [ f_1(x_1) ] ---------\
[ x_2 ] ---------> [ f_2(x_2) ] ---------> Sum (+) ---> Prediction
[ x_3 ] ---------> [ f_3(x_3) ] ---------/
Taking this concept into deep learning, Agarwal et al. introduced Neural Additive Models (NAMs). Instead of trees, NAMs train a dedicated, independent neural network for each individual feature. The outputs of these single-feature networks are then linearly summed. This allows the model to learn highly complex, non-linear shape functions via deep learning while strictly preventing feature interactions, preserving total functional transparency.
Concept Bottlenecks and Prototypical Routing
For complex data like images or text, univariate feature isolation is impossible because raw pixels or tokens carry no meaning in isolation. Here, the strategy shifts to forcing the network to route its decisions through human-understandable intermediate concepts or examples.
This is the core mechanism of Concept Bottleneck Models (CBMs). Instead of mapping an input directly to a final label ($X \to Y$), the model is split into two parts: it first maps the input to a set of human-defined concepts ($X \to C$), and then uses a transparent linear layer to predict the label from those concepts ($C \to Y$). To avoid the massive computational cost of training these models from scratch, Dominici et al. developed AnyCBMs. This framework maps the internal hidden embeddings of any pre-trained black-box network onto human-understandable concepts post-hoc, granting manual intervention capabilities without sacrificing the underlying model's accuracy.
Another approach relies on prototypical reasoning: explaining a decision by pointing to representative examples. Li et al. introduced HyperPg, which structures the latent space of deep networks using Gaussian distributions on a hypersphere to serve as interpretable prototypes. When the model classifies an input, it calculates proximity to these geometric prototypes, allowing humans to audit the exact conceptual evidence driving the conclusion.
Distilling Black Boxes into Symbolic Logic
When a black-box model must be deployed due to legacy constraints or raw performance requirements, researchers use distillation to extract human-readable logic rules from its predictions.
A foundational tool for local explanations is SHAP, introduced by Lundberg et al., which uses game-theoretic Shapley values to fairly attribute a model's specific prediction to its input features. Building on local attribution methods, Shakerin et al. developed LIME-FOLD, an algorithm that automatically translates complex ensemble models (like XGBoost) into concise, non-monotonic logic programs. By analyzing local feature importances across a dataset, it induces symbolic rules that humans can read like a checklist.
Similarly, Lee et al. proposed a game-theoretic framework for "functional transparency." Instead of explaining a model after the fact, their framework penalizes a neural network during training if its local behavior deviates from a simple, verifiable class of functions, forcing the complex model to remain locally interpretable by design.
Why Explaining "Why" Improves "What"
Intuitively, adding interpretability constraints feels like it should always degrade performance because it restricts the model's optimization space. However, empirical evidence shows that forcing a model to be transparent can actually improve its accuracy and robustness.
Opaque models are notorious for exploiting spurious correlations—such as using the background texture of an image to classify an animal, or using hospital-specific formatting to diagnose a disease. When models are structurally forced to justify their reasoning, they are constrained to focus on causal, robust features.
For instance, in autonomous driving, Xu et al. demonstrated that forcing an end-to-end neural network to explicitly identify "action-inducing" objects and output textual explanations of its reasoning directly led to more accurate driving commands. By aligning the internal representations with human-verifiable logic, the model avoided catastrophic edge-case errors that plague purely unconstrained black boxes. Interpretability and performance, when engineered correctly, are not opposing forces—they are complementary properties of robust generalization.
Key Foundational Work
Intelligible models for classification and regression (2012) — contributed 1 concept elements to this direction.
Towards Transparent Control of Large and Complex Systems (1999) — contributed 1 concept elements to this direction.