Abstract
The success of deep neural nets heavily relies on their ability to encode complex relations between their input and their output. While this property serves to fit the training data well, it also obscures the mechanism that drives prediction. This study aims to reveal hidden concepts by employing an intervention mechanism that shifts the predicted class based on discrete variational autoencoders. An explanatory model then visualizes the encoded information from any hidden layer and its corresponding intervened representation. By the assessment of differences between the original representation and the intervened representation, one can determine the concepts that can alter the class, hence providing interpretability. We demonstrate the effectiveness of our approach on CelebA, where we show various visualizations for bias in the data and suggest different interventions to reveal and change bias.
Introduction
Deep neural networks (DNNs) have revolutionized fields from computer vision to natural language processing by learning intricate patterns in high-dimensional data. However, their internal decision-making processes remain largely opaque, earning them the reputation of "black boxes." While local interpretability methods like saliency maps provide insights into which input features influence a prediction, they often fail to convey the high-level semantic concepts—such as gender, age, or specific facial features—that drive global model behavior. For instance, a classifier might predict "attractiveness" based on systemic biases rather than aesthetic features, a nuance that pixel-level maps might obscure. This paper addresses the need for global, concept-based explanations by proposing a framework that maps internal representations to a discrete latent space, allowing for direct intervention and visualization of the model's underlying logic.
Methodology
The proposed framework operates in three distinct stages to decode the internal logic of a pre-trained classifier. First, the continuous representation from a chosen hidden layer of the classifier is passed through a Discrete Variational Autoencoder (DVAE). This process maps the complex feature space into a set of binary bits ($z$), where each bit is hypothesized to represent a distinct high-level concept used by the model.
Second, an explanatory generator ($g$) is trained to reconstruct the input image from these discrete bits. To ensure that the generator remains faithful to the classifier's internal reasoning, the authors introduce a novel regularization term based on functional Fisher information. This alignment ensures that the visual output $x'$ reflects the information actually utilized by the classifier, rather than merely producing a generic realistic image.
Finally, the framework enables "interventions" by flipping specific bits in the latent space to create a counterfactual representation $\hat{z}$. By observing how flipping a bit changes the classifier's prediction (e.g., from "Class A" to "Class B") and comparing the resulting reconstructed images ($x'$ vs $\hat{x}'$), researchers can visually identify the specific concept that triggered the classification shift.
Results and Discussion
The effectiveness of the approach was evaluated using the CelebA dataset, focusing on attributes such as "Smiling," "Attractive," and "Big Lips." The method demonstrated a high intervention success rate, with bit-flipping successfully reversing the classifier's prediction in 92% to 98% of cases. Crucially, the reconstructed representations maintained high classification accuracy—within approximately 1% of the original model—proving that the discretization process did not lose critical information.
A key insight revealed by the visualizations is the "laziness" of classifiers, which often rely on spurious correlations. For the "attractive" task, the model's internal concept was found to be heavily entangled with "young female" features. When the "attractive" bit was flipped to its negative state, the reconstructed images consistently shifted toward older male figures, exposing a systemic bias in the training data or model architecture. Furthermore, the use of a discrete latent space proved superior for causal inference compared to continuous spaces, as binary bits provide a clear, finite search space for counterfactual explanations.
Conclusion
This study presents a robust framework for interpreting deep learning models through latent space intervention. By aligning a discrete latent space with a classifier's internal representations, the method provides clear, visual explanations of high-level concepts and uncovers hidden biases. Future work will explore the scalability of this approach to non-visual domains and the evolution of concepts across different layers of the neural network.
Summary updated