TL;DR
Attention mechanisms are widely assumed to make deep learning models inherently interpretable, but this paper demonstrates that high classification accuracy does not guarantee a model is focusing on the right features. By introducing a novel framework called Selective Dependence Classification (SDC), the authors objectively quantify interpretability and show that sparsity-inducing activation functions like Sparsemax can significantly improve a model's alignment with ground-truth explanations.
What Problem Does This Paper Address?
Deep learning practitioners frequently use attention weights as a window into a model's "reasoning"—for instance, showing a heatmap of where an image captioning model looks when it outputs the word "dog" (as seen in Figure 1). However, these claims of interpretability are usually qualitative and subjective.
The core challenge this paper addresses is the lack of an objective, quantitative framework to evaluate whether attention models actually attend to the features responsible for a label. The authors investigate a troubling phenomenon: Can a model achieve near-perfect accuracy while looking at completely irrelevant parts of the input?
Methods
To study this rigorously, the authors introduce two new concepts:
- Selective Dependence Classification (SDC): A specialized problem setting where an input instance is a "mosaic" of $m$ distinct segments. Crucially, the true label depends only on a single "foreground" segment, while the remaining $m-1$ segments are background noise. The training data contains no labels indicating which segment is the foreground.
- Focus-Classify Attention Model (FCAM): A simplified attention architecture tailored for SDC tasks. It consists of a focus network that scores each segment's likelihood of being the foreground, and a classification network that predicts the final label from the attention-weighted average of the segments.
Using this setup, they define an objective interpretability metric called Focus True (FT): the fraction of test instances where the focus network successfully assigns its highest score to the true foreground segment.
They leverage this framework to evaluate several methods designed to encourage attention sparsity:
- Entropy Regularization: Adding an entropy penalty to the loss function to discourage diffuse attention weights.
- Sparsemax & Spherical Softmax: Alternative activation functions to standard Softmax that naturally map low probabilities to exact zeros.
- Hard Attention: A stochastic approach that samples a single segment rather than taking a weighted average.
Datasets
The paper evaluates these methods across three distinct settings:
- MS COCO: Used for an initial case study to verify that standard attention models align significantly better with ground-truth object bounding boxes than random baselines.
- Synthetic SDC Dataset: A low-dimensional (2D) synthetic dataset where foreground and background segments are drawn from distinct Gaussian distributions, allowing for easy visualization of model behavior.
- CIFAR-SDC: A semi-synthetic dataset where each instance is a mosaic of 9 stitched CIFAR-10 images. One image is the foreground (drawn from car, plane, bird), and the other eight are background images drawn from the remaining seven CIFAR-10 classes.
Key Insights and Ideas
- Three Error Modes of Attention: The authors prove that accurate models can fail to be interpretable via three distinct training behaviors:
- First Error Mode: The focus network mistakes a specific foreground class for background, but the classifier is flexible enough to fix the skewed distribution and still predict accurately.
- Second Error Mode ("Two Wrongs Make a Right"): A flawed focus network and a flawed classifier cancel each other's errors out to achieve high accuracy, completely misleading anyone looking at the attention weights.
- Third Error Mode: The classification network is powerful enough to identify the foreground class even when the attention vector is completely uniform (flat), meaning the focus network never actually learns to "attend."
- Sparsity Drives Truth: Forcing the attention vector to be sparse compels the focus network to make a definitive choice, which naturally guides it toward the true foreground segment during optimization.
Key Results
On the semi-synthetic CIFAR-SDC dataset, all models achieved high test accuracy (~95%), but their interpretability varied drastically:
- Standard Softmax achieved a baseline 79.77% FT interpretability.
- Sparsemax and Spherical Softmax significantly boosted interpretability to 91.43% FT and 91.07% FT respectively, without sacrificing any classification accuracy.
- Late-Layer Aggregation: Averaging features at the last hidden layer of the focus network (rather than the raw input layer) consistently improved interpretability across the board (e.g., standard Softmax jumped from 79.77% to 86.21% FT).
Limitations and Future Work
While the SDC framework provides a clean, mathematical sandbox for quantifying interpretability, it is inherently simplified. Real-world data rarely features a clean, binary division between a single foreground feature and independent background noise; instead, labels often depend on complex interactions between multiple parts of an input.
Additionally, while Hard Attention provides strong theoretical interpretability, it caused a massive drop in accuracy (down to 74.45% on CIFAR-SDC) and significantly increased computational training costs. The authors note that exploring more complex data distributions and identifying further attention error modes remain open directions for future research.
Summary updated