Caption-Driven Explainability: Probing CNNs for Bias via CLIP

Patrick Koller, Amil V. Dravid, Guido M. Schuster, Aggelos K. KatsaggelosarXiv 2025

Abstract

Robustness has become one of the most critical problems in machine learning (ML). The science of interpreting ML models to understand their behavior and improve their robustness is referred to as explainable artificial intelligence (XAI). One of the state-of-the-art XAI methods for computer vision problems is to generate saliency maps. A saliency map highlights the pixel space of an image that excites the ML model the most. However, this property could be misleading if spurious and salient features are present in overlapping pixel spaces. In this paper, we propose a caption-based XAI method, which integrates a standalone model to be explained into the contrastive language-image pre-training (CLIP) model using a novel network surgery approach. The resulting caption-based XAI model identifies the dominant concept that contributes the most to the models prediction. This explanation minimizes the risk of the standalone model falling for a covariate shift and contributes significantly towards developing robust ML models.

Introduction

Deep learning models, particularly Convolutional Neural Networks (CNNs), are often criticized as "black boxes." While they achieve high accuracy in controlled environments, they frequently fail in real-world scenarios due to covariate shift—a phenomenon where the model relies on spurious correlations in the training data that do not hold true in the wild. A classic example is a digit classifier that learns to identify the number "5" based on its red color rather than its shape, simply because all "5s" in the training set were red.

Existing Explainable AI (XAI) tools, such as Grad-CAM, utilize saliency maps to highlight which pixels influenced a prediction. However, these maps only show where a model is looking, not what it is seeing. If a red digit is highlighted, a saliency map cannot distinguish whether the model is focusing on the color or the geometry. This paper introduces a method for concept-level explanations to bridge this gap, allowing researchers to interrogate the internal logic of a model using natural language.

Methodology: Network Surgery

The researchers developed a technique called Network Surgery to transplant the internal features of a standard CNN into the rich conceptual space of the CLIP (Contrastive Language-Image Pre-training) model.

  1. The Standalone Model: The process begins with a standard ResNet-50 trained on a specific task, such as classifying colored MNIST digits.
  2. Activation Matching: The team identifies corresponding layers between the standalone model and CLIP’s internal image encoder (a modified ResNet-52) by calculating correlation coefficients between their activation maps.
  3. The Surgery: Approximately 16.9% of CLIP’s activation maps are surgically replaced with the corresponding maps from the standalone model.
  4. Concept Probing: Images are passed through this hybrid "surgically altered" encoder to generate embeddings that reflect the standalone model's internal logic.
  5. Cosine Similarity: These embeddings are compared against text prompts (e.g., "a photo of a red digit" vs. "a photo of a digit with value five") using CLIP’s text encoder. The prompt with the highest cosine similarity reveals the dominant concept driving the prediction.
The Network Surgery framework integrating a standalone encoder with CLIP.

Experimental Setup and Datasets

The method was validated using Colored MNIST, a modified version of the classic handwritten digits dataset. In the biased training set, specific digits were tied to specific colors (e.g., '5' was always red). A separate "real-world" test set was created where colors were assigned randomly to test the model's robustness. The standalone ResNet-50 was initially pre-trained on ImageNet before being fine-tuned on this colored digit task.

Results and Discussion

The caption-driven approach successfully exposed hidden biases that traditional metrics missed. In the biased model, the tool identified that the prediction was 100% reliant on the "color" concept. Even when the model correctly identified a "5," the XAI tool revealed it was doing so for the wrong reasons.

Comparison of biased vs. unbiased model explanations.

When the model was "de-biased" by training on grayscale images, the tool quantitatively tracked the improvement. The probability of the "shape" concept significantly increased, while the "color" concept probability dropped. This confirms that the model had shifted its internal logic from superficial shortcuts to robust features.

Conclusion

By leveraging CLIP as a "universal translator" for visual features, this method provides a human-readable way to audit AI models before deployment. While currently limited by the architectural constraints of the "surgery" (swapping only 17% of maps), the approach demonstrates immense potential for high-stakes fields like medical imaging. Ensuring a model identifies a lesion based on its pathology rather than incidental markers (like a ruler in the frame) is a vital step toward trustworthy and robust artificial intelligence.

Summary updated

Abstract

Robustness has become one of the most critical problems in machine learning (ML). The science of interpreting ML models to understand their behavior and improve their robustness is referred to as explainable artificial intelligence (XAI). One of the state-of-the-art XAI methods for computer vision problems is to generate saliency maps. A saliency map highlights the pixel space of an image that excites the ML model the most. However, this property could be misleading if spurious and salient features are present in overlapping pixel spaces. In this paper, we propose a caption-based XAI method, which integrates a standalone model to be explained into the contrastive language-image pre-training (CLIP) model using a novel network surgery approach. The resulting caption-based XAI model identifies the dominant concept that contributes the most to the models prediction. This explanation minimizes the risk of the standalone model falling for a covariate shift and contributes significantly towards developing robust ML models.
BibTeX
@misc{koller2025caption,
  author = {Patrick Koller and Amil V. Dravid and Guido M. Schuster and Aggelos K. Katsaggelos},
  title = {{Caption-Driven Explainability: Probing CNNs for Bias via CLIP}},
  year = {2025},
  eprint = {2510.22035},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.22035},
}
Figure