Self-Interpretable Model with Transformation Equivariant Interpretation

Yipei Wang, Xiaoqian Wang

The Fragility of "Black Box" Explanations

Deep learning models are notoriously opaque. To understand why a model classified an image as a "dog," researchers typically rely on post-hoc methods like Grad-CAM, which generate heatmaps after the model is already trained. However, these methods are often fragile: a slight rotation or shift of the input image might cause the explanation to jump to a completely different, irrelevant part of the image.

Furthermore, there is a long-standing belief in AI that you must choose between accuracy and interpretability. Simple models (like decision trees) are easy to understand but perform poorly on complex tasks, while deep neural networks are highly accurate but inscrutable. A new framework called SITE (Self-Interpretable model with Transformation Equivariant Interpretation) aims to bridge this gap by creating a high-performing model that is "born" interpretable.

The SITE Framework: Built-in Interpretability

Unlike standard models that pass features directly into a final classification layer, SITE uses a multi-step, self-interpretable process. By enforcing "transformation equivariance," the model ensures that if you rotate or shift an image, its explanation (the heatmap) transforms in exactly the same way.

SITE Architecture

The architecture operates through four main steps:

  1. Feature Extraction: A standard backbone (like ResNet) extracts latent features from an image.
  2. Prototype Generation: A generative model takes those features and creates dynamic "prototypes" for each possible class. These represent what the model expects to see given the specific input, allowing it to handle the massive variety found in real-world images.
  3. Self-Interpretation: The model calculates its prediction by taking the inner product of the features and the prototypes. The interpretation is generated via a Hadamard product, highlighting exactly which features contributed to the match.
  4. Equivariant Regularization: During training, a specialized loss function forces the model's interpretations to be equivariant. If the input image is transformed (e.g., rotated), the model is penalized if its internal prototypes and resulting interpretations don't reflect that exact transformation.

Key Insights: Breaking the Trade-off

The authors argue that if an explanation doesn't move with the object it's explaining, it isn't truly reflecting the model's predictive mechanism. Equivariance isn't just a bonus; it's a requirement for a faithful explanation.

Crucially, SITE proves that interpretability doesn't have to cost accuracy. In several tests, SITE actually outperformed the black-box models it was based on. The authors attribute this to the regularization process, which acts as a form of "informed" data augmentation.

Performance and Robustness

Evaluated on benchmarks like MNIST, CIFAR-10, and BAM, SITE demonstrated both high accuracy and superior interpretability.

  • Superior Accuracy: On CIFAR-10, SITE using a ResNet-18 backbone achieved 89.2% accuracy, outperforming the standard black-box ResNet-18 (86.2%).
  • Self-Consistency: The authors introduced a "Self-Consistency Score" to quantify how well an explanation follows an object across transformations. SITE achieved a score of 0.8860, significantly higher than popular post-hoc methods like Guided Backpropagation (0.7830) or Gradient-based methods (0.7174).
Qualitative Comparison of Explanations

Qualitatively, visualizations show that SITE consistently highlights the actual silhouette of objects (like the shape of a ship) even when those objects are heavily rotated or shifted. In contrast, post-hoc methods often produce noisy, disconnected, or entirely derailed heatmaps when faced with the same transformations.

Limitations and Future Directions

While promising, SITE has a few limitations. It currently uses bilinear upsampling to move from the latent space back to image pixels, which is a rough approximation that might lose fine-grained detail. Additionally, the current study focused primarily on rotation and translation; future work is needed to test more complex distortions like scaling, shearing, or non-rigid deformations. Finally, while highly effective for computer vision, this transformation-equivariant approach has yet to be tested on non-visual data like natural language or genomic sequences.

Summary updated

Abstract

With the proliferation of machine learning applications in the real world, the demand for explaining machine learning predictions continues to grow especially in high-stakes fields. Recent studies have found that interpretation methods can be sensitive and unreliable, where the interpretations can be disturbed by perturbations or transformations of input data. To address this issue, we propose to learn robust interpretation through transformation equivariant regularization in a self-interpretable model. The resulting model is capable of capturing valid interpretation that is equivariant to geometric transformations. Moreover, since our model is self-interpretable, it enables faithful interpretations that reflect the true predictive mechanism. Unlike existing self-interpretable models, which usually sacrifice expressive power for the sake of interpretation quality, our model preserves the high expressive capability comparable to the state-of-the-art deep learning models in complex tasks, while providing visualizable and faithful high-quality interpretation. We compare with various related methods and validate the interpretation quality and consistency of our model.
BibTeX
@inproceedings{wang2021self,
  author = {Yipei Wang and Xiaoqian Wang},
  title = {{Self-Interpretable Model with Transformation Equivariant Interpretation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=FD-pxYFucRb},
}
Figure