Evaluating Concept Explanations for CNNs Under Adversarial Image Transformations

Ugochukwu Ejike Akpudo, Yongsheng Gao, Andrew Lewis, Edwin Kwadwo Tenagyei, Yi Liao, Jun ZhouInternational Conference on Intelligent and Innovative Computing Applications 2024 · DOI

Abstract

Concept-based explainers for convolutional neural networks (CNNs) provide human-understandable explanations by revealing what the CNN sees, rather than merely indicating where it looked. However, their performance is limited by the reducer at its core and adversarial attacks. Although CNN classification performance may be enhanced by some image transformations in small amounts whereas intense image transformations can cause noticeable variations to CNN predictions, it is uncertain how explainers perform in such cases. This paper investigates the performance of state-of-the-art concept-based explainers at different levels of adversarial attacks for the first time. We achieve this by exploring different image transformations as adversarial attacks, including Gaussian noise, elastic transform, rotation, and contrast on the ILSVRC2012 dataset. Our study shows that image transformation techniques altering only image coordinates have little impact on classifier and explainer performance, whereas methods modifying image pixels, such as elastic transform and contrast, significantly affect performance, akin to introducing Gaussian noise. Our work underscores the significance of scrutinizing explainers during their development and adoption for CNNs.

Introduction

As Convolutional Neural Networks (CNNs) are increasingly integrated into high-stakes domains such as medical diagnostics and autonomous driving, the need for transparency becomes paramount. Traditional "attribution" methods, like Grad-CAM, identify which pixels contributed to a decision but fail to explain the underlying logic. Concept-based explainers address this by identifying high-level, human-understandable patterns—such as "stripes" or "ears"—that drive model outputs.

However, these explainers rely on dimensionality reduction techniques (reducers) that may be susceptible to adversarial perturbations. This study investigates whether common image transformations that affect CNN classification also compromise the integrity of the generated explanations.

Methodology

The research evaluates two prominent concept-based frameworks: Invertible Concept-based Explanations (ICE) and Concept Recursive Activation FacTorization (CRAFT). Both frameworks utilize Non-negative Matrix Factorization (NMF) to decompose CNN activation maps into interpretable concepts.

Overview of the concept-based explanation framework, showing the path from input images through the encoder and reducer to the latent concept space.

The robustness of these frameworks was tested using a pre-trained ResNet50 model on the ILSVRC2012 (ImageNet) dataset. Four types of adversarial transformations were applied to the input:

  1. Gaussian Noise: Pixel-level random interference.
  2. Elastic Transform: Localized image warping.
  3. Rotation: Coordinate shifts without pixel value modification.
  4. Contrast: Adjustments to pixel intensity ranges.

Performance was primarily measured using Fidelity ($Z$), which assesses how accurately the explainer captures the model's internal logic.

Results and Analysis

Spatial vs. Pixel-Level Robustness

A key finding is the stark difference in robustness between spatial and pixel-level transformations. Transformations that merely recalculate coordinates, such as Rotation, had negligible impact on explainer fidelity. Even at extreme rotation angles, fidelity remained near baseline levels (~97-99%).

In contrast, transformations that modify pixel values—Gaussian Noise, Contrast, and Elastic Transform—led to significant performance degradation. For instance, increasing Gaussian noise ($\sigma$) from 0.1 to 1.0 caused the ICE explainer's fidelity for the "Australian Kelpie" class to drop from 94.89% to 36.71%.

Concept Degradation

The study observed that as noise levels increase, images from distinct classes become statistically similar, a phenomenon termed the "inter-class similarity trap." This prevents the explainer from isolating the discriminative features necessary for coherent concept formation.

Visual prototypes for 'Kelpie' and 'Chihuahua' classes, showing the concepts and their associated fidelity (Z) and weight (W) scores.

Visually, the "prototypes" (representative image patches for a concept) become increasingly blurred and lose semantic meaning as adversarial intensity rises. This visual decay directly correlates with the quantitative drop in fidelity scores.

Conclusion

This research highlights a critical vulnerability in state-of-the-art concept-based explainers: while they are resilient to spatial changes, they are highly sensitive to pixel-level alterations. These findings suggest that the "reducers" at the heart of these explainers are a primary point of failure under adversarial conditions. Future work should focus on developing more robust reduction techniques and standardized benchmarks for evaluating the reliability of Explainable AI (XAI) tools in real-world, noisy environments.

Summary updated

Abstract

Concept-based explainers for convolutional neural networks (CNNs) provide human-understandable explanations by revealing what the CNN sees, rather than merely indicating where it looked. However, their performance is limited by the reducer at its core and adversarial attacks. Although CNN classification performance may be enhanced by some image transformations in small amounts whereas intense image transformations can cause noticeable variations to CNN predictions, it is uncertain how explainers perform in such cases. This paper investigates the performance of state-of-the-art concept-based explainers at different levels of adversarial attacks for the first time. We achieve this by exploring different image transformations as adversarial attacks, including Gaussian noise, elastic transform, rotation, and contrast on the ILSVRC2012 dataset. Our study shows that image transformation techniques altering only image coordinates have little impact on classifier and explainer performance, whereas methods modifying image pixels, such as elastic transform and contrast, significantly affect performance, akin to introducing Gaussian noise. Our work underscores the significance of scrutinizing explainers during their development and adoption for CNNs.
BibTeX
@inproceedings{akpudo2024evaluating,
  author = {Ugochukwu Ejike Akpudo and Yongsheng Gao and Andrew Lewis and Edwin Kwadwo Tenagyei and Yi Liao and Jun Zhou},
  title = {{Evaluating Concept Explanations for CNNs Under Adversarial Image Transformations}},
  booktitle = {International Conference on Intelligent and Innovative Computing Applications},
  year = {2024},
  doi = {10.59200/iconic.2024.006},
}
Figure