Stable Vision Concept Transformers for Medical Diagnosis

Lijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, Di Wang

TL;DR

The paper introduces the Stable Vision Concept Transformer (SVCT), a novel framework that brings robust, human-understandable explanations to medical image diagnosis without sacrificing accuracy. By fusing raw Vision Transformer (ViT) features with automated concept layers and stabilizing them via Denoised Diffusion Smoothing, SVCT ensures that AI explanations remain reliable and faithful even when images contain noise or perturbations.

What Problem Does This Paper Address?

Deep learning models excel at medical image classification, but their "black-box" nature makes clinical adoption difficult; physicians need to know why a model made a specific diagnosis. Concept Bottleneck Models (CBMs) try to solve this by forcing the network to predict intermediate, human-understandable concepts (e.g., "diffuse irregular pigmentation" or "dark lung color") before outputting the final diagnosis.

However, standard CBMs suffer from two major flaws:

  1. The Utility Gap: They usually perform worse than standard models because they discard rich, intrinsic image features in favor of a limited set of text-based concepts.
  2. Explanation Instability: They are highly sensitive to minor input perturbations. Adding a tiny, imperceptible amount of noise to an image can completely scramble the model's predicted concepts, rendering the explanation unfaithful and dangerous for medical deployment.
Fig. 1: An example of VCT framework on OCT2017 dataset [29]. The leftmost figure displays the input image, while the adjacent one on the left shows the con- cept output without perturbations. In contrast, the figure on the right presents the concept output after applying input perturbations, resulting in noticeable changes.

Methods

The authors tackle these issues in two stages, building up to the Stable Vision Concept Transformer (SVCT):

  1. Vision Concept Transformer (VCT): To bridge the utility gap, VCT uses a Vision Transformer (ViT) backbone. Instead of relying only on concepts for the final prediction, VCT fuses the high-level concept features with the raw patch embeddings from the ViT backbone. To keep the model "label-free" (avoiding expensive manual concept labeling), they use GPT-3 to generate a broad set of medical concepts and leverage CLIP-Dissect to align image features with these concepts.
  2. Denoised Diffusion Smoothing (DDS): To solve the instability problem, the authors upgrade VCT to SVCT by integrating DDS into the token embeddings. DDS injects randomized Gaussian noise into the image embeddings and then uses a denoising diffusion probabilistic model to smooth them out. The authors provide a formal mathematical proof showing that this process guarantees the top-$k$ predicted concepts remain stable under input perturbations.
Fig. 2: Overview of our Stable Vision Concept Transformer (SVCT) model.

Datasets

The framework was comprehensively evaluated across four distinct medical image modalities:

  • HAM10000: 10,015 dermatoscopic skin lesion images.
  • Covid19-CT: 746 lung CT scan images.
  • BloodMNIST: A benchmark dataset containing images of individual normal blood cells categorized into eight types.
  • OCT2017: 84,484 Optical Coherence Tomography images for retinal macular disease diagnosis.

Key Insights and Ideas

  • Feature Fusion is Essential: Discarding raw image features for pure concept bottlenecks hurts accuracy. Fusing raw visual embeddings with concept vectors acts as a powerful regularizer, preserving performance while maintaining interpretability.
  • Diffusion as a Smoothing Mechanism: While diffusion models are typically celebrated for image generation, this paper uniquely repurposes them as a mathematical smoothing tool to defend the interpretability of a transformer against adversarial or natural noise.
  • Interactive Test-Time Intervention: Because the concept layer is explicit, medical experts can interact with the model in real-time. If the model misidentifies a concept, a doctor can manually "correct" it, and the model will dynamically update its final diagnosis accurately.

Key Results

  • Accuracy Preservation: On the HAM10000 dataset without perturbations, standard uninterpretable ViT achieves 99.13% accuracy. Label-Free CBM drops significantly to 93.61%, whereas SVCT maintains a near-lossless 99.05%.
  • Robustness Under Attack: When subjected to adversarial perturbations ($\rho_u = 10/255$), the baseline Label-Free CBM's accuracy plummets to 88.70%. SVCT retains a highly robust 97.24% accuracy.
  • Explanation Stability: Using metrics like the Concept Faithfulness Score (CFS) and Concept Perturbation Cosine Similarity (CPCS), SVCT consistently demonstrated vastly superior stability in its concept weights pre- and post-perturbation compared to all baseline models.

Limitations and Future Work

While SVCT provides strong theoretical and empirical stability, the authors note a few limitations:

  • Gaussian Noise Assumption: The mathematical proofs and smoothing techniques are optimized around Gaussian noise. Real-world clinical environments may introduce other complex artifacts or domain shifts that require further testing.
  • Human Evaluation: The concepts are generated and filtered automatically via LLMs and vision-language models. True clinical utility will require extensive future studies involving direct evaluation and collaboration with medical professionals to judge the quality and relevance of the explanations.

Summary updated

Abstract

Transparency is a paramount concern in the medical field, prompting researchers to delve into the realm of explainable AI (XAI). Among these XAI methods, Concept Bottleneck Models (CBMs) aim to restrict the model's latent space to human-understandable high-level concepts by generating a conceptual layer for extracting conceptual features, which has drawn much attention recently. However, existing methods rely solely on concept features to determine the model's predictions, which overlook the intrinsic feature embeddings within medical images. To address this utility gap between the original models and concept-based models, we propose Vision Concept Transformer (VCT). Furthermore, despite their benefits, CBMs have been found to negatively impact model performance and fail to provide stable explanations when faced with input perturbations, which limits their application in the medical field. To address this faithfulness issue, this paper further proposes the Stable Vision Concept Transformer (SVCT) based on VCT, which leverages the vision transformer (ViT) as its backbone and incorporates a conceptual layer. SVCT employs conceptual features to enhance decision-making capabilities by fusing them with image features and ensures model faithfulness through the integration of Denoised Diffusion Smoothing. Comprehensive experiments on four medical datasets demonstrate that our VCT and SVCT maintain accuracy while remaining interpretable compared to baselines. Furthermore, even when subjected to perturbations, our SVCT model consistently provides faithful explanations, thus meeting the needs of the medical field.
BibTeX
@inproceedings{hu2025stable,
  author = {Lijie Hu and Songning Lai and Yuan Hua and Shu Yang and Jingfeng Zhang and Di Wang},
  title = {{Stable Vision Concept Transformers for Medical Diagnosis}},
  booktitle = {ECML/PKDD},
  year = {2025},
  url = {https://openreview.net/forum?id=4VfqT2gwrH},
}
Figure