CE-FAM: Concept-Based Explanation via Fusion of Activation Maps

Michihiro Kuroki, Toshihiko Yamasaki

Abstract

Although saliency maps highlight important regions for image classification, their semantic meaning is often ambiguous. We propose Concept-based Explanation via Fusion of Activation Maps (CE-FAM), a novel framework that decomposes AI predictions into human-understandable concepts. CE-FAM identifies where concepts are located and quantifies their contribution to the final prediction by leveraging Vision-Language Model (VLM) knowledge and fusing multi-layer activation maps. Our method enables zero-shot inference for arbitrary concepts and introduces the Normalized Region Accuracy (NRA) metric for more precise evaluation of concept localization.

Introduction: Beyond Saliency Maps

Most modern AI interpretability tools rely on saliency maps (e.g., Grad-CAM) to highlight "important" pixels. However, these maps are often ambiguous—a highlighted region on a bird might signify color, shape, or texture, leaving the "why" to user interpretation. Concept-based explanations address this by using human-interpretable terms. Existing methods, however, often require expensive annotated datasets, struggle to map concepts distributed across multiple layers, or fail to simultaneously show localization and contribution.

Overview of CE-FAM: Identifying concept regions and quantifying their contributions to the final classification of a "jay".

Methodology: The CE-FAM Framework

CE-FAM addresses these limitations through a branched architecture and a multi-layer fusion strategy.

VLM Mimicry and Zero-Shot Learning

Instead of requiring manual labels for every concept, CE-FAM adds a small branch to a standard image classifier. This branch is trained to mimic the embeddings of a pre-trained Vision-Language Model (VLM) like CLIP. By aligning the classifier's internal representations with the VLM's semantic space, the model can understand arbitrary text concepts (e.g., "striped," "wooden") without specific training data.

Multi-Layer Activation Map Fusion

Concepts are rarely localized in a single neuron or layer. Low-level layers might capture color, while high-level layers capture complex object parts. CE-FAM extracts activation maps from all layers and fuses them using the gradients of the predicted concept scores as weights. This produces cleaner, more accurate localizations than methods relying on single-channel "dissection."

The CE-FAM architecture: A branched network extracts and fuses activation maps from multiple layers, trained to mimic VLM embeddings.

Quantifying Contribution

To determine a concept's importance, CE-FAM calculates an "impact score." By masking the identified concept region and measuring the resulting drop in the final classification probability, the method quantifies exactly how much a specific concept swayed the model's decision.

Evaluation and Results

The researchers evaluated CE-FAM on benchmarks including ImageNet, Broden (for localization), and Animals with Attributes 2 (AwA2) for zero-shot inference.

Normalized Region Accuracy (NRA)

Standard Intersection over Union (IoU) is often unsuitable for saliency-style explanations that lack hard boundaries. The authors introduced Normalized Region Accuracy (NRA), which compares predicted regions against ground-truth masks while accounting for the probability of random pixel highlighting.

Key Performance Metrics

  • Localization: On the Broden dataset with a ResNet50, CE-FAM achieved an average NRA of 0.361, outperforming baselines like CLIP-Dissect (0.334) and WWW (0.278).
  • Hit Rate: On ImageNet-S, CE-FAM reached a hit rate of 0.744, significantly higher than the 0.674 achieved by the next best method.
  • Versatility: The framework proved effective across different architectures, including both CNNs (ResNet) and Transformers (ViT).

Discussion and Future Work

CE-FAM demonstrates that concepts are distributed across network layers and can be effectively extracted via VLM alignment. While powerful, the method's vocabulary is currently bounded by the underlying VLM's capabilities. Future work will focus on automating the selection of "necessary and sufficient" concept sets and extending the framework to non-visual domains.

Summary updated

Abstract

Although saliency maps can highlight important regions to explain the reasoning behind image classification in artificial intelligence (AI), the meaning of these regions is left to the user's interpretation. In contrast, conceptbased explanations decompose AI predictions into humanunderstandable concepts, clarifying their contributions. However, few methods can simultaneously reveal what concepts an image classifier learns, which regions are associated with them, and how they contribute to predictions. We propose a novel concept-based explanation method, Concept-based Explanation via Fusion of Activation Maps (CE-FAM). It employs a branched network that shares activation maps with an image classifier and learns to mimic the embeddings of a Vision and Language Model (VLM). The branch network predicts concepts in an image, and their corresponding regions are represented by a weighted sum of activation maps, with weights given by the gradients of the concept prediction scores. Their contributions are quantified based on their impact on the image classification score. Our method provides a general framework for identifying the concept regions and their contributions while leveraging VLM knowledge to handle arbitrary concepts without requiring an annotated dataset. Furthermore, we introduce a novel evaluation metric to assess the accuracy of the concept regions. Our qualitative and quantitative evaluations demonstrate our method outperforms existing approaches and excels in zero-shot inference for unseen concepts.
BibTeX
@inproceedings{kuroki2025ce,
  author = {Michihiro Kuroki and Toshihiko Yamasaki},
  title = {{CE-FAM: Concept-Based Explanation via Fusion of Activation Maps}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure