Abstract
Neural networks are part of daily-life decision-making, including in high-stakes settings where understanding and transparency are key. Saliency maps have been developed to gain understanding into which input features neural networks use for a specific prediction. Although widely employed, these methods often result in overly general saliency maps that fail to identify the specific information that triggered the classification. In this work, we suggest a framework that allows to incorporate attributions across classes to arrive at saliency maps that actually capture the class-relevant information. On established benchmarks for attribution methods, including the grid-pointing game and randomization-based sanity checks, we show that our framework heavily boosts the performance of standard saliency map approaches. It is, by design, agnostic to model architectures and attribution methods and now allows to identify the distinguishing and shared features used for a model prediction.
Introduction
In the field of Explainable AI (XAI), attribution maps—heatmaps highlighting pixels that drive a specific classification—are essential tools for transparency. However, existing methods often produce "blurry" logical results, highlighting all salient objects in a scene rather than the specific features unique to the target class. For instance, in an image containing both a zebra and a bison, a standard attribution map for "Zebra" might highlight both animals because both contribute positively to raw evidence scores.
The core issue lies in how attributions are typically computed: relative to a single "target" class logit in isolation. In practice, neural network decisions are made in the softmax layer, where evidence for one class is weighed against all others. Direct attribution of the softmax output is often uninformative due to the "vanishing gradient" problem in confident predictions. This paper introduces the Attribution Lens (AL), a refinement framework that reintroduces class competition to produce sharper, more class-specific explanations.
The Attribution Lens Framework
The Attribution Lens (AL) is a model-agnostic, training-free refinement that transforms standard saliency maps into class-relevant distributions. The framework operates in three primary steps:
- Multi-class Attribution: Instead of a single map, AL computes base attribution maps for the target class and a set of "distractor" classes (typically the top-k most probable classes).
- Pixel-wise Competition: At each pixel, a softmax is applied across these different class attributions. This creates a local distribution identifying which class "owns" the importance of that specific pixel.
- Refinement: The original attribution map is scaled by this local dominance score, effectively dampening pixels where the target class does not clearly win the competition.
By moving the contrastive nature of classification from the final output layer down to individual pixels, AL allows the model to reveal highly specific, fine-grained explanations that were previously obscured.
Experimental Evaluation
The researchers evaluated AL across 18 different XAI methods and various architectures, including ResNet-50 and Vision Transformers (ViTs). Evaluation benchmarks included:
- Localization Accuracy: Using the "Grid Pointing Game" on ImageNet and PartImageNet, and alignment with segmentation masks on MS-COCO.
- Faithfulness: Insertion and Deletion tests, which measure how quickly model confidence changes when pixels identified as "important" are added or removed.
- Sanity Checks: Cascading randomization of model weights to ensure attributions are truly dependent on learned parameters.
Results and Discussion
The quantitative improvements provided by AL are substantial. On ResNet-50, localization metrics like Region Attribution (RA) and Intersection over Union (IoU) improved by an average of 53% and 37%, respectively. The impact on Vision Transformers was even more pronounced, with IoU and F1 scores more than doubling in several tests. AL effectively acts as a denoiser for ViTs, filtering out background noise and focusing on relevant object tokens.
Qualitatively, AL enables the differentiation of closely related classes. For example, it can highlight the specific ear shape distinguishing a Lynx from a Cougar, rather than merely highlighting the general feline form. In "Insertion" tests, AL-refined pixels were found to be significantly more critical to the model's actual decision, improving the Area Under the Curve (AUC) by 8–11% for CNNs.
Conclusion
The Attribution Lens framework demonstrates that class-specific information is often "hidden in plain sight" within existing neural network models. By leveraging class competition, AL significantly enhances the precision and faithfulness of saliency maps across diverse architectures and attribution methods. While it introduces some computational overhead—linear with the number of competing classes—much of the backward pass can be optimized. Future work may explore more sophisticated methods for selecting adversarial classes to provide even deeper insights into model behavior.
Summary updated