Black Box Explanation by Learning Image Exemplars in the Latent Feature Space

Riccardo Guidotti, Anna Monreale, Stan Matwin, Dino PedreschiarXiv 2020 · arXiv

TL;DR

ABELE (Adversarial Black box Explainer generating Latent Exemplars) provides human-interpretable explanations for image classifiers by generating realistic "exemplars" and "counter-exemplars." By operating in a latent feature space learned via an Adversarial Autoencoder, it avoids the "unrealistic noise" problem of traditional explainers and offers intuitive visual evidence for why a model made a specific decision.

What Problem Does This Paper Address?

As deep learning models become standard in sensitive fields like medicine and finance, their "black box" nature becomes a liability. We need to know why a model classified an X-ray as "malignant" or a digit as "4."

Existing methods generally fall into two camps, both with flaws:

  1. Saliency Maps: These produce heatmaps of important pixels. However, they are often model-specific (only working for certain neural networks) and can be noisy or unstable.
  2. Model-Agnostic Local Explainers (like LIME): These perturb the input image to see how the prediction changes. The problem? Their perturbations often create "meaningless" images (e.g., images with random gray blocks) that don't look like real data, leading to unreliable local approximations of the model's behavior.

ABELE aims to bridge this gap by generating a local neighborhood of meaningful images that respect the underlying data distribution.

Methods

The core innovation of ABELE is the use of an Adversarial Autoencoder (AAE) to manage the explanation process in a compressed, latent feature space rather than raw pixel space.

The process follows these steps:

  1. Latent Encoding: An image $x$ is compressed into a latent representation $z$ using the AAE's encoder.
  2. Neighborhood Generation: ABELE generates a set of synthetic neighbors in the latent space around $z$. Because it uses an AAE, the discriminator ensures these neighbors, when decoded, look like realistic images.
  3. Local Rule Extraction (llore): A decision tree is trained on this latent neighborhood to mimic the black box's behavior locally. This tree provides symbolic rules (e.g., "if feature A > 0.5, then class is '9'").
  4. Exemplar Generation: Using these rules, ABELE samples the latent space to find "exemplars" (images that look like $x$ and share its class) and "counter-exemplars" (minimal changes that would flip the class).
  5. Visual Explanation: It produces a saliency map by calculating the median difference between the original image and its exemplars.
Fig. 1. Left: Adversarial Autoencoder architecture: the encoder turns the image x into its latent representation z, the decoder re-builds an approximation ex of x from z, and the discriminator identifies if a randomly generated latent instance h can be considered valid or not. Right: Discriminator and Decoder (disde) module: input is a randomly generated latent instance h and, if it is considered valid by the discriminator, it returns it together with its decompressed version eh.
The llore module extracts decision rules and counter-factuals from the latent neighborhood.

Datasets

The authors evaluated ABELE on three standard image classification benchmarks:

  • MNIST: Grayscale handwritten digits.
  • Fashion-MNIST: Grayscale images of clothing items (Zalando's dataset).
  • CIFAR-10: Color images of 10 different object classes (airplanes, cats, etc.).

They tested explanations against two types of black boxes: Random Forests (RF) and Deep Neural Networks (DNN).

Key Insights and Ideas

  • The Power of Latent Space: Perturbing pixels directly is inefficient and often results in "garbage" data. By perturbing latent features, ABELE captures semantic relationships (like the curve of a '9' or the neck of a 'boot') that raw pixels ignore.
  • Counter-factuals as Explanations: Knowing what makes a '4' a '4' is useful, but seeing exactly how a '4' could "morph" into a '9' (a counter-factual) provides a much deeper level of human understanding.
  • Meaningful Saliency: Unlike gradient-based methods that might highlight scattered pixels, ABELE’s saliency maps highlight contiguous, semantically relevant areas because they are derived from realistic variations of the image.

Key Results

ABELE was compared against LIME and several saliency-based methods (like Integrated Gradients and Occlusion).

  • Fidelity: ABELE’s local decision tree mimicked the black box more accurately and with much lower variance than LIME across all datasets.
  • Stability and Coherence: ABELE proved significantly more robust to noise. While other methods' saliency maps changed drastically with slight input perturbations (salt-and-pepper noise), ABELE’s explanations remained consistent.
  • Relevance: In "masking" experiments (hiding the most important pixels to see if the prediction flips), ABELE was more effective at identifying the truly critical parts of an image, especially in complex datasets like CIFAR-10.

Limitations and Future Work

  • Data Specificity: Currently, the method is constrained to image data. The authors suggest extending it to tabular and text data in the future.
  • AAE Dependency: The quality of the explanation depends heavily on the quality of the Adversarial Autoencoder. If the AAE fails to capture the data distribution well, the exemplars won't be meaningful.
  • Computational Cost: Training an AAE for every new dataset adds an extra step compared to simple perturbation methods.
  • Future Directions: The authors plan to apply ABELE to medical imaging (like fMRI) and conduct human-in-the-loop studies to quantitatively measure how much these explanations actually help users make better decisions.

Summary updated

Abstract

We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by "morphing" into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
BibTeX
@misc{guidotti2020black,
  author = {Riccardo Guidotti and Anna Monreale and Stan Matwin and Dino Pedreschi},
  title = {{Black Box Explanation by Learning Image Exemplars in the Latent Feature Space}},
  year = {2020},
  eprint = {2002.03746},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.03746},
}

Loading related content...

Figure