Hidden in Plain Sight -- Class Competition Focuses Attribution Maps

Nils Philipp Walter, Jilles Vreeken, Jonas FischerICML 2026 · OpenReview

TL;DR

This paper introduces "Attribution Lens" (AL), a simple post-processing technique that refines existing attribution methods by considering class competition. By analyzing attributions across multiple classes, AL significantly improves their specificity and fine-grainedness, making them more useful for understanding neural network decisions without requiring model retraining.

What Problem Does This Paper Address?

Attribution methods are crucial for understanding how neural networks make decisions by highlighting important input features. However, a common issue is that these attributions are often unspecific, highlighting both relevant and irrelevant features, making it difficult to discern what truly drives a prediction. The authors argue that this problem stems from the standard practice of computing attributions based on a single class's logit, which ignores the inherent contrastive nature of the softmax layer used for final predictions. This leads to diffuse and sometimes misleading explanations.

Methods

The core contribution is the Attribution Lens (AL), a plug-and-play refinement that operates on existing attribution maps. Instead of focusing on a single class's logit, AL computes attributions for a set of relevant classes and then normalizes these attributions at each pixel using a spatial softmax. This creates a "pixel-wise attribution distribution" that reflects how much each pixel contributes to the competition between classes. The AL then refines the original attribution map for the target class by multiplying it with this class-competition distribution. This process effectively "focuses" the attribution, highlighting features that are discriminative for the target class while downplaying those shared with or relevant to other classes. The authors explore three strategies for selecting the set of classes: predefined sets, top-k most probable classes, and best-vs-worst classes.

Datasets

The paper uses several standard datasets for evaluation:

  • ImageNet: Used for the grid-pointing game benchmark.
  • PartImageNet: A subset of ImageNet with part annotations, also used for the grid-pointing game.
  • MS-COCO: Used for evaluating localization ability.
  • Retinal OCT dataset: Used for a case study in medical imaging.

Key Insights and Ideas

  • Class competition is key: The paper's central insight is that the "hidden in plain sight" solution to unspecific attributions lies in considering how classes compete with each other, mirroring the softmax function's role in final predictions.
  • Logit-based attributions are limiting: Directly attributing to a single class's logit discards crucial information about inter-class relationships, leading to less informative maps.
  • AL is model-agnostic and training-free: The refinement can be applied to any existing attribution method and any neural network architecture without retraining, making it highly practical.
  • Existing methods already contain rich signals: The authors demonstrate that even standard attribution methods, when viewed through the lens of class competition, reveal class-specific information that was previously obscured.

Key Results

  • Significant improvement in specificity and fine-grainedness: AL consistently refines attributions, making them more object-specific, concept-specific, and fine-grained across various attribution methods and architectures.
  • Up to 2x improvement on benchmarks: AL improves benchmark metrics (like Region Attribution, IoU, and F1) for 18 attribution methods across 7 architectures by up to 2x.
  • Strong performance on localization tasks: AL significantly enhances localization ability, particularly on the Grid Pointing Game and MS-COCO datasets.
  • Robustness to randomization: AL-augmented methods show improved stability under network randomization tests, indicating better faithfulness to the model's learned representations.
  • Qualitative improvements: Visual examples demonstrate AL's ability to sharpen attributions, isolate discriminative features, and reveal class-specific structures, even in challenging medical imaging cases.

Limitations and Future Work

  • Limitations of sanity checks: The authors acknowledge that randomization-based sanity checks have limitations, as noted by Binder et al. (2023), and can sometimes be passed by methods that don't truly reflect model understanding.
  • Superpixel-based perturbation challenges: For perturbation-based methods using superpixels, AL's gains on MS-COCO were modest, and sometimes even negative for F1 and IoU, due to superpixels not always aligning with object extents in cluttered scenes.
  • Future work: The paper suggests that understanding the interplay between training objectives (like BCE) and attribution quality is an interesting avenue, as BCE might implicitly encourage learning class-specific features that AL can then better reveal. Further exploration of AL's theoretical properties and its application to other domains like natural language processing are also implied.

Summary updated

Abstract

Attribution methods reveal which input features a neural network uses for a prediction, adding transparency to their decisions. A common problem is that these attributions seem unspecific, highlighting both important and irrelevant features. We revisit the common attribution pipeline and observe that using logits as attribution target is a main cause of this phenomenon. We show that the solution is in plain sight: considering distributions of attributions over multiple classes using existing attribution methods yields specific and fine-grained attributions. On common benchmarks, including the grid-pointing game and randomization-based sanity checks, this improves the ability of 18 attribution methods across 7 architectures up to $2\times$, agnostic to model architecture.
BibTeX
@inproceedings{walter2026hidden,
  author = {Nils Philipp Walter and Jilles Vreeken and Jonas Fischer},
  title = {{Hidden in Plain Sight -- Class Competition Focuses Attribution Maps}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=DSZ36ji3Y6},
}
Figure