Rethinking Interpretation: Input-Agnostic Saliency Mapping of Deep Visual Classifiers

Naveed Akhtar, Mohammad Amir Asim Khan Jalwana

Introduction

Saliency mapping is a cornerstone of explainable AI (XAI), providing heatmaps that highlight influential pixels in a model's prediction. However, traditional methods are fundamentally input-specific: they explain why a model classified a particular image as a "Panda," rather than what the model understands about the "Panda" category as a whole. This dependency often leads to maps that merely highlight object boundaries, failing to capture the model's internal logic or detect hidden vulnerabilities like backdoors.

Comparison between proposed input-agnostic saliency and conventional input-specific methods.

This paper introduces a novel input-agnostic saliency mapping technique. By decoupling the visualization from specific input images, the method reveals the high-level geometric features a model associates with a learned concept, offering a more holistic interpretation of deep visual classifiers.

The Input-Agnostic Approach

The core limitation of conventional saliency is its "weak dependence" on the classifier. If a method produces similar maps for both a flawed model and a perfect theoretical one, it isn't truly explaining the specific model's behavior. Input-agnostic mapping addresses this by generating visualizations directly from the model's internal gradients without requiring a reference dataset.

Methodology

The proposed algorithm iteratively constructs a map from a neutral "seed" (e.g., a black image) through three main stages:

  1. Gradient Nudging: Random data points are "nudged" toward the local minima of the model’s loss surface for a target class.
  2. Integration and Projection: These gradient directions are accumulated and projected onto a bounded $\ell_2$-ball. This constraint forces disparate features to "collaborate," resulting in coherent geometric patterns rather than isolated high-intensity pixels.
  3. Refinement: To suppress non-perceptual noise, a weighting matrix (inspired by GradCAM) is applied, highlighting the structural features the model prioritizes.

Experimental Results

The method was evaluated on VGG-16, ResNet-50, and DenseNet-121 architectures pre-trained on ImageNet.

Qualitative Analysis

The generated maps for various ImageNet classes demonstrate that the model learns distinct, human-interpretable textures and shapes. For instance, the "Honeycomb" map captures hexagonal patterns, while the "Mosque" map visualizes dome-like structures.

Input-agnostic visualizations for various ImageNet concepts, showing the geometric features learned by the model.

Quantitative Fidelity

To measure how well these maps represent generic features, the authors introduced the Model-score ($M_{score}$). A map generated for a specific model (e.g., VGG-16) is fed into other classifiers. The results showed high cross-architecture consistency:

  • Intra-architecture: VGG-16 maps achieved a score of 0.99 when tested on other VGG-16 instances.
  • Inter-architecture: Maps from ResNet-50 maintained high scores (~0.94) when evaluated by DenseNet-121, suggesting that different models converge on similar geometric concepts.

Security Applications: Backdoor Detection

A significant advantage of input-agnostic mapping is its ability to expose "Trojan" backdoors. In compromised models, the map for a target class will often reconstruct the hidden trigger pattern (e.g., a specific logo) used to fool the classifier. This allows for the detection of malicious triggers without prior knowledge of their appearance, a task nearly impossible with standard input-specific saliency.

Conclusion and Future Work

Input-agnostic saliency mapping provides a powerful new lens for auditing deep learning models. While the current refinement stage uses external operators and the computational cost is higher than traditional methods (14–19 minutes per map), the ability to visualize "pure" model concepts and detect security threats represents a major advancement in model interpretability. Future research will focus on identifying non-visible adversarial triggers and further reducing reliance on external refinement operators.

Summary updated

Abstract

Saliency methods provide post-hoc model interpretation by attributing input features to the model outputs. Current methods mainly achieve this using a single input sample, thereby failing to answer input-independent inquiries about the model. We also show that input-specific saliency mapping is intrinsically susceptible to misleading feature attribution. Current attempts to use `general' input features for model interpretation assume access to a dataset containing those features, which biases the interpretation. Addressing the gap, we introduce a new perspective of input-agnostic saliency mapping that computationally estimates the high-level features attributed by the model to its outputs. These features are geometrically correlated, and are computed by accumulating model's gradient information with respect to an unrestricted data distribution. To compute these features, we nudge independent data points over the model loss surface towards the local minima associated by a human-understandable concept, e.g., class label for classifiers. With a systematic projection, scaling and refinement process, this information is transformed into an interpretable visualization without compromising its model-fidelity. The visualization serves as a stand-alone qualitative interpretation. With an extensive evaluation, we not only demonstrate successful visualizations for a variety of concepts for large-scale models, but also showcase an interesting utility of this new form of saliency mapping by identifying backdoor signatures in compromised classifiers.
BibTeX
@inproceedings{akhtar2023rethinking,
  author = {Naveed Akhtar and Mohammad Amir Asim Khan Jalwana},
  title = {{Rethinking Interpretation: Input-Agnostic Saliency Mapping of Deep Visual Classifiers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25089},
}
Figure