ProtoAttend: Attention-Based Prototypical Learning

Sercan Ömer Arik, Tomas PfisterJMLR 2020 · OpenReview

TL;DR

This paper introduces ProtoAttend, a novel machine learning method that makes decisions based on a small set of "prototypical" examples. By using an attention mechanism, ProtoAttend achieves high-quality interpretability, state-of-the-art confidence estimation, and robust out-of-distribution detection, all without sacrificing model accuracy.

What Problem Does This Paper Address?

Deep neural networks, while powerful, are often treated as "black boxes." This lack of interpretability hinders trust and deployment in critical applications where understanding why a decision was made is crucial. Existing methods for explaining these models are often "post-hoc," meaning they analyze a trained model after the fact, which can be limited. The authors aim to create inherently interpretable models that are designed for transparency from the ground up. Specifically, they address the challenge of making complex models understandable by grounding their decisions in a few representative examples (prototypes) and providing reliable confidence scores.

Methods

ProtoAttend is built upon the concept of prototypical learning, where decisions are made by comparing an input to a set of learned prototypes (representative examples from the training data). The core innovation is the attention mechanism that ProtoAttend uses to dynamically select and weight these prototypes.

Here's a breakdown of the key components:

  • Encoder: A trainable neural network that transforms input data into representations (keys, queries, and values). This encoder is shared across inputs and prototype candidates to learn a common representation space.
  • Relational Attention: This mechanism uses the keys and queries generated by the encoder to compute attention coefficients (weights) between an input and each prototype candidate. These weights indicate how relevant each prototype is to the current input. The authors explore both softmax and sparsemax normalization functions for these weights, with sparsemax encouraging sparsity (i.e., selecting fewer prototypes).
  • Decision Making: The final decision is made by a linear combination of the values of the selected prototypes, weighted by the attention coefficients. This allows the model to leverage the information from relevant prototypes.
  • Training Objectives: The authors propose several training objectives that balance conventional supervised learning with the principles of prototypical learning, including encouraging prototype agreement and controlling sparsity.
  • Confidence Scoring: A novel confidence score is derived from the agreement among the labels of the high-weight prototypes. If prototypes with high weights all agree on a class, the model is more confident.

ProtoAttend is designed to be model-agnostic, meaning it can be integrated into various neural network architectures, including pre-trained models, by adding a few extra layers for attention.

Datasets

The paper evaluates ProtoAttend on a variety of datasets across different modalities:

  • Image: MNIST, Fashion-MNIST, CIFAR-10, Fruits, ISIC Melanoma, Animals with Attributes.
  • Text: DBPedia.
  • Tabular: Income (Adult Census Income).

Key Insights and Ideas

  • Attention for Prototype Selection: Using attention to dynamically select and weight prototypes based on input similarity is a powerful way to achieve interpretability. It moves beyond static prototype selection methods.
  • Sparse Prototypes for Interpretability: Sparsemax attention and sparsity regularization allow the model to rely on a very small number of prototypes, making the decision process much easier for humans to follow.
  • Confidence from Prototype Agreement: The idea that confidence can be derived from the consensus of relevant prototypes is a novel and effective approach, outperforming traditional methods in some cases.
  • Unified Framework for Multiple Benefits: ProtoAttend achieves interpretability, confidence estimation, and out-of-distribution detection within a single, coherent framework, demonstrating its versatility.
  • Minimal Performance Degradation: The method achieves these interpretability benefits with very little to no loss in classification accuracy compared to standard baseline models.

Key Results

  • Interpretability: ProtoAttend provides high-quality sample-based interpretability, outputting the most relevant prototypes for a given decision. Figures 4, 5, 6, 10, 11, 12, 13, and 14 showcase these visually.
  • Confidence Estimation: The method achieves state-of-the-art confidence estimation, with higher confidence scores correlating strongly with higher accuracy (Figure 3, Table 1).
  • Out-of-Distribution Detection: ProtoAttend demonstrates strong performance in detecting out-of-distribution samples, achieving an AUC of 0.838 on CIFAR-10 (Figure 9).
  • Robustness to Label Noise: The method shows improved accuracy on noisy datasets compared to baseline and dropout methods (Table 4).
  • Accuracy: ProtoAttend maintains comparable overall accuracy to baseline models across various datasets, with minimal degradation (Table 2). For example, on MNIST, it achieves 99.69% accuracy with sparsemax attention, very close to the baseline's 99.70%.

Limitations and Future Work

  • Computational Cost: While inference is efficient, training ProtoAttend can be slower than conventional supervised learning due to the need to process candidate prototypes. The authors suggest data and model parallelism as solutions.
  • Prototype Quality: The paper acknowledges scenarios that can lead to low prototype quality, such as a lack of related samples in the candidate database, perceptual differences between human and encoder feature interpretation, high intra-class variability, and imperfect encoders (Appendix F).
  • Future Work: The authors suggest further research into information-theoretic metrics for prototype quality and using discriminative neural networks for this purpose. They also mention that the decoupling between encoder, attention, and decision blocks could be further improved.

Summary updated

Abstract

We propose a novel inherently interpretable machine learning method that bases decisions on few relevant examples that we call prototypes. Our method, ProtoAttend, can be integrated into a wide range of neural network architectures including pre-trained models. It utilizes an attention mechanism that relates the encoded representations to samples in order to determine prototypes. Protoattend yields superior results in three high impact problems without sacrificing accuracy of the original model: (1)it enables high-quality interpretability that outputs samples most relevant to the decision-making (i.e. a sample-based interpretability method); (2) it achieves state of the art confidence estimation by quantifying the mismatch across prototype labels; and (3) it obtains state of the art in distribution mismatch detection. All these can be achieved with minimal additional test time and a practically viable training time computational cost.
BibTeX
@article{arik2020protoattend,
  author = {Sercan Ömer Arik and Tomas Pfister},
  title = {{ProtoAttend: Attention-Based Prototypical Learning}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
  url = {https://openreview.net/forum?id=rGbFjLFF9K},
}
Figure