Born Identity Network: Multi-way Counterfactual Map Generation to Explain a Classifier's Decision

Kwanseok Oh, Jee Seok Yoon, Heung-Il SukarXiv 2020 · arXiv

Introduction

Deep learning models, while achieving state-of-the-art performance across various domains, often operate as "black boxes." This lack of interpretability is a significant hurdle in high-stakes fields like medicine. Existing Explainable AI (XAI) techniques generally fall into two categories: attribution-based methods (like Grad-CAM) and counterfactual methods. Attribution methods highlight important pixels but often produce blurry results and only answer why a specific label was chosen. Counterfactual methods answer "what would change the outcome?" but are typically limited to binary or single-way transitions.

The Born Identity Network (BIN) addresses these limitations by providing a post-hoc framework for generating "multi-way" counterfactual maps. BIN demonstrates how an image would need to change to be classified as a different category—such as transforming a normal brain scan into one indicating Alzheimer’s—while strictly preserving the original sample's identity.

Methodology: The Born Identity Network

The BIN framework consists of two primary modules designed to work in a post-hoc fashion, allowing it to explain pre-trained classifiers without requiring retraining.

Counterfactual Map Generator (CMG)

The CMG is a conditional generative model, specifically a U-Net architecture, that takes an input image $x$ and a target label $y$. It synthesizes a sparse "counterfactual map" $M$. When this map is added to the original image, it creates a synthesized image $\tilde{x} = x + M$ that the classifier should recognize as belonging to class $y$.

Target Attribution Network (TAN)

The TAN is the pre-trained classifier being explained. During the training of the BIN, the TAN's weights are frozen. It serves as a guide for the CMG, ensuring that the generated modifications effectively shift the classification toward the desired target label.

Overview of the BIN framework, illustrating the interaction between the Counterfactual Map Generator (CMG), the Target Attribution Network (TAN), and the Discriminator.

The "Born Identity" Constraint

A key innovation is the Counterfactual Map Loss ($L_{map}$). To ensure the synthesized image remains recognizable as the original subject, the authors use $L_1$ and $L_2$ regularization. This forces the modifications to be as minimal and sparse as possible, focusing changes only on the "Regions of Interest" (ROIs) necessary to flip the classification.

Experimental Evaluation

The model was evaluated on three datasets: a synthetic 3D Shapes dataset, the MNIST handwritten digit dataset, and the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset containing 3D MRI scans.

Key Results and Visual Explanations

In clinical validation using the ADNI dataset, BIN successfully localized known biomarkers of Alzheimer’s disease, such as ventricular enlargement and cortical thinning. Quantitatively, BIN outperformed existing methods like VAGAN and traditional attribution methods (Integrated Gradients, DeepLIFT) in terms of Normalized Cross-Correlation (NCC) against ground-truth longitudinal data.

Visual examples of multi-way counterfactual map generation for MNIST digits and ADNI brain MRI scans, showing the input, the generated map, and the resulting synthesized image.

On the MNIST dataset, the synthesized images achieved a target classification accuracy of 99.12%, demonstrating that the counterfactual maps effectively "fooled" the classifier into seeing the intended target digit while maintaining the original digit's structural identity.

Discussion and Insights

BIN's ability to support multi-way stratification is particularly valuable for modeling disease progression. Unlike binary explanations, BIN can visualize intermediate stages (e.g., Normal $\rightarrow$ Mild Cognitive Impairment $\rightarrow$ Alzheimer’s), providing a spectrum of visual evidence. Furthermore, its post-hoc flexibility means it can be applied to any high-performance model to enhance interpretability without sacrificing predictive power.

Limitations and Future Work

The authors identified potential biases introduced by activation functions like ReLU, which might skew model outputs. Additionally, while the results align with medical knowledge, further clinical validation is required to determine if these hypothetical "counterfactual realities" can reliably assist clinicians in early-stage diagnosis. The computational intensity of generating high-resolution 3D maps also remains a challenge for real-time applications.

Summary updated

Abstract

There exists an apparent negative correlation between performance and interpretability of deep learning models. In an effort to reduce this negative correlation, we propose a Born Identity Network (BIN), which is a post-hoc approach for producing multi-way counterfactual maps. A counterfactual map transforms an input sample to be conditioned and classified as a target label, which is similar to how humans process knowledge through counterfactual thinking. For example, a counterfactual map can localize hypothetical abnormalities from a normal brain image that may cause it to be diagnosed with a disease. Specifically, our proposed BIN consists of two core components: Counterfactual Map Generator and Target Attribution Network. The Counterfactual Map Generator is a variation of conditional GAN which can synthesize a counterfactual map conditioned on an arbitrary target label. The Target Attribution Network provides adequate assistance for generating synthesized maps by conditioning a target label into the Counterfactual Map Generator. We have validated our proposed BIN in qualitative and quantitative analysis on MNIST, 3D Shapes, and ADNI datasets, and showed the comprehensibility and fidelity of our method from various ablation studies.
BibTeX
@misc{oh2020born,
  author = {Kwanseok Oh and Jee Seok Yoon and Heung-Il Suk},
  title = {{Born Identity Network: Multi-way Counterfactual Map Generation to Explain a Classifier's Decision}},
  year = {2020},
  eprint = {2011.10381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.10381},
}
Figure