Latent Space Explanation by Intervention

Itai Gat, Guy Lorberbom, Idan Schwartz, Tamir Hazan

Abstract

The success of deep neural nets heavily relies on their ability to encode complex relations between their input and their output. While this property serves to fit the training data well, it also obscures the mechanism that drives prediction. This study aims to reveal hidden concepts by employing an intervention mechanism that shifts the predicted class based on discrete variational autoencoders. An explanatory model then visualizes the encoded information from any hidden layer and its corresponding intervened representation. By the assessment of differences between the original representation and the intervened representation, one can determine the concepts that can alter the class, hence providing interpretability. We demonstrate the effectiveness of our approach on CelebA, where we show various visualizations for bias in the data and suggest different interventions to reveal and change bias.

Introduction

Despite their success, deep neural networks (DNNs) are notoriously difficult to interpret. Most existing explainability (XAI) methods, such as saliency maps, focus on local feature relevance—highlighting which pixels were important for a specific prediction. However, knowing where a model looked doesn't necessarily explain what it saw or what high-level concepts it utilized. Furthermore, counterfactual explanations typically require manual annotation of concepts, which is expensive and limited by human priors. This paper proposes a method to discover and visualize the internal concepts a model has learned in an unsupervised manner.

Methodology

The authors propose a two-stage framework to generate visual, concept-level explanations by intervening in the latent space of a pre-trained classifier.

  1. Discrete Latent Encoding: The system focuses on a hidden layer of the classifier. A Discrete Variational Autoencoder (DVAE) is trained to map this layer's representation into a set of boolean latent variables ($z \in \{0, 1\}^n$). These bits represent the "concepts" encoded by the model.
  2. Intervention & Visualization: To understand the semantic meaning of a specific bit, the system flips its value (an intervention) and observes the resulting change in the classifier's output. An explanatory generator ($g$) is then used to reconstruct images from both the original and intervened hidden representations.
  3. Functional Information Regularization: To ensure the generator visualizes exactly what the classifier "sees," the authors maximize the shared functional information between the classifier and the generator using a regularization term based on functional Fisher information.
The proposed architecture showing the flow from input $x$ through the classifier to the discrete latent space $z$, and finally to the explanatory generator $g$.

Key Insights: Unsupervised Concept Discovery and Bias Detection

By flipping bits in the latent space and observing the classifier's prediction flip, the method establishes a causal link between the visualized concept and the model's decision. This allows for the discovery of influential concepts without explicit supervision.

For example, in the CelebA dataset, the system identified that a classifier for "attractiveness" was heavily biased toward younger faces. Similarly, if a "smiling" classifier is trained on a dataset where mostly women smile, the intervention to make a face "smile" might also make the face look more feminine in the visualization, revealing that the model has conflated gender with the target attribute.

Examples of interventions for 'Attractive' and 'Big nose' tasks. Flipping discrete latent bits changes the model's prediction and reveals the visual concepts associated with those labels.

Experimental Results

The effectiveness of the approach was evaluated on several CelebA classification tasks:

  • Intervention Success: Flipping the discovered latent bits changed the classifier's prediction with high frequency: 98.86% for "Smiling" and 97.08% for "Attractive."
  • Faithful Reconstruction: The discretization process preserved essential information; reconstructed representations maintained high accuracy (e.g., 91.80% for smiling) compared to the original representations (92.10%).
  • Regularization Impact: Shared information regularization significantly improved alignment. On the "Attractive" task, the cosine similarity between the classifier and generator information vectors increased from 0.37 to 0.80.

Conclusion and Future Work

This framework provides a powerful tool for auditing deep learning models for hidden biases and understanding their decision-making processes. While it currently requires human interpretation of the generated images, it automates the discovery of the underlying concepts. Future work includes extending this to non-image domains like natural language processing and improving the clarity of visualizations using more advanced generative models like Diffusion models.

Summary updated

Abstract

The success of deep neural nets heavily relies on their ability to encode complex relations between their input and their output. While this property serves to fit the training data well, it also obscures the mechanism that drives prediction. This study aims to reveal hidden concepts by employing an intervention mechanism that shifts the predicted class based on discrete variational autoencoders. An explanatory model then visualizes the encoded information from any hidden layer and its corresponding intervened representation. By the assessment of differences between the original representation and the intervened representation, one can determine the concepts that can alter the class, hence providing interpretability. We demonstrate the effectiveness of our approach on CelebA, where we show various visualizations for bias in the data and suggest different interventions to reveal and change bias.
BibTeX
@inproceedings{gat2022latent,
  author = {Itai Gat and Guy Lorberbom and Idan Schwartz and Tamir Hazan},
  title = {{Latent Space Explanation by Intervention}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=PuYH8UEEfjp},
}
Figure