Modern deep learning models often operate as "black boxes," making decisions based on high-dimensional patterns that are difficult for humans to interpret. To make these models more trustworthy, researchers have developed techniques for explainability, fairness, and adversarial robustness. A recurring theme across these domains is the use of auxiliary generative models to ensure that interventions—such as changing a feature or generating a counterfactual—remain "on-manifold," or consistent with real-world data distributions.
However, this reliance creates a fundamental dependency: the utility and reliability of the resulting explanations or fairness guarantees are strictly bounded by the fidelity and coverage of the underlying generative architecture.
The Manifold Constraint Problem
Many traditional interpretability methods, such as saliency maps or Integrated Gradients, work by perturbing input features and observing the change in a model's output. A major flaw in this approach is that arbitrary perturbations often push data points into "off-manifold" regions—areas of the feature space that the model never saw during training. As Melamed and Caruana argue, these off-manifold gradients often point toward adversarial noise rather than human-understandable features.
To solve this, researchers use generative models to constrain operations to the data manifold. For example, Zaher et al. propose Manifold Integrated Gradients (MIG), which replaces straight-line paths with geodesics along a manifold learned by a generator. Similarly, Lei et al. use diffusion models to guide attribution paths, ensuring intermediate states remain realistic. In both cases, the explanation's accuracy is directly tied to how well the generative model captures the true data structure.
Explainability through Synthesis
Generative models are particularly powerful for creating "what-if" scenarios, known as counterfactual explanations. Instead of just highlighting pixels, these methods show how an input would need to change to reach a different classification.
- In-filling and Baselines: Chang and Goldenberg introduced FIDO, which uses a conditional generator to fill in masked regions of an image with plausible pixels. This avoids the artifacts created by simple blurring. Similarly, Shih et al. use GANs to generate class-specific baselines, allowing for "one-vs-one" explanations that highlight features differentiating two specific categories.
- Latent Space Interventions: Methods like CLUE, proposed by Antoran et al., and LAPACE, proposed by Junqi Jiang, search for counterfactuals in the latent space of a VAE. By moving through this structured space, they ensure the resulting explanations are both diverse and plausible.
The intuition here is that a good explanation should look like a real example. If the generative model is weak, the "explanation" might be a blurry or nonsensical image, which O'Shaughnessy and Canal note can lead to a lack of precision in high-stakes domains.
Fairness and Robustness as Generative Tasks
The dependency on generative fidelity extends to fairness and security. To train fairer models, researchers often need to see how a model would behave if a sensitive attribute (like race or gender) were changed.
Li et al. and Ma et al. both utilize generative models to create "counterfactual" or "antidote" data. These synthetic samples allow models to learn representations that are invariant to sensitive traits. However, as the authors point out, if the generator fails to preserve identity or introduces its own biases, the resulting "fair" model may actually perform worse or inherit the generator's flaws.
In the realm of security, Hsu et al. show that detecting adversarial attacks can be framed as measuring the mutual information between an input and its generative reconstruction. If the generator is high-quality, it will fail to reconstruct adversarial noise, making the attack obvious. If the generator is poor, it might fail to reconstruct even natural images, leading to false positives.
The Fidelity Bottleneck
The core insight across these 57 papers is that generative fidelity is the bottleneck for downstream trust. Whether it is Zhang et al. using optimal generative flows for Shapley values or Gong et al. decoding visual thoughts from fMRI scans, the logic remains the same:
- The Task: We need to explore the data manifold (for explanations, fairness, or exploration).
- The Tool: We use a generative model (VAE, GAN, Diffusion, or Flow) to approximate that manifold.
- The Failure Mode: Any error in the generative model—blurriness, mode collapse, or lack of coverage—propagates directly into the final result.
As Osada et al. demonstrate in the context of latent adversarial training, high-fidelity, lossless models like Normalizing Flows often provide a much clearer signal than traditional VAEs. This suggests that as generative modeling continues to improve, our ability to explain and secure "black box" models will scale alongside it, provided we account for the inherent biases of the generators themselves.
Key Foundational Work
Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives (2018) — contributed 1 concept elements to this direction.
On The Utility of Conditional Generation Based Mutual Information for Characterizing Adversarial Subspaces (2018) — contributed 1 concept elements to this direction.
Generating Natural Adversarial Examples (2018) — contributed 1 concept elements to this direction.