Stable Vision Concept Transformers for Medical Diagnosis

Lijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, Di Wang

Introduction

Deep learning models have demonstrated remarkable success in medical imaging, yet their "black box" nature remains a significant barrier to clinical adoption. In healthcare, a diagnosis without a clear rationale is difficult for clinicians to trust or verify. To bridge this gap, researchers have turned to Concept Bottleneck Models (CBMs). These models map input images to human-understandable concepts (e.g., "irregular pigmentation" or "dark lung color") before making a final prediction.

However, traditional CBMs face two critical hurdles: a "performance tax" where accuracy drops because the model is restricted to a limited set of concepts, and "explanation instability," where minor, invisible noise in an image can cause the model's reported reasoning to change drastically. This paper introduces the Stable Vision Concept Transformer (SVCT), a framework designed to provide both state-of-the-art accuracy and mathematically guaranteed explanation stability.

The SVCT Framework

The authors propose a two-stage evolution of the standard Vision Transformer (ViT) architecture to address the shortcomings of existing interpretable models.

1. Vision Concept Transformer (VCT)

To eliminate the performance degradation seen in traditional CBMs, the authors developed the VCT. Instead of forcing the model to rely exclusively on a bottleneck of concepts, the VCT uses a label-free approach—leveraging GPT-3 to generate medical concepts and CLIP to align them with image features. These interpretable concept features are then fused (concatenated) with the raw, high-dimensional feature embeddings from the ViT backbone. This allows the model to retain the rich information necessary for high-accuracy diagnosis while still outputting the importance of specific human-readable concepts.

2. Denoised Diffusion Smoothing (DDS)

To solve the problem of brittle explanations, the authors introduced Denoised Diffusion Smoothing (DDS). During the inference process, the model adds random Gaussian noise to the image's token embeddings and then utilizes a diffusion model to denoise them before concept extraction. This process acts as a mathematical shock absorber. The authors provide formal proofs demonstrating that under this smoothing process, the top-$k$ most important concepts are guaranteed to remain stable even under bounded input perturbations, ensuring the model's "reasoning" is faithful and robust.

Experimental Results

The SVCT was evaluated across four diverse medical imaging datasets: HAM10000 (dermatology), Covid19-CT (lung CT), BloodMNIST (blood cells), and OCT2017 (retinal scans).

Performance and Robustness

The SVCT matched or exceeded the performance of non-interpretable Vision Transformers. On the HAM10000 dataset, the SVCT achieved an accuracy of 99.05%, significantly outperforming the baseline Label-Free CBM at 93.61%.

When subjected to heavy input noise ($\rho_u = 10/255$), the SVCT's accuracy remained high at 97.24%, while the baseline CBM's performance plummeted to 88.70%.

Explanation Stability

The study utilized the Concept Faithfulness Score (CFS) to measure how much concept rankings shifted under noise (where lower scores indicate higher stability). Under heavy noise, the SVCT maintained a CFS of 0.1725, whereas the baseline model scored 0.4027. This confirms that the SVCT’s explanations remain consistent even when the input data is imperfect.

Clinical Impact and Human-in-the-Loop AI

One of the most promising aspects of the SVCT is its support for test-time intervention. Because the model's internal logic is exposed through human-readable concepts, a doctor can manually correct a misidentified concept (e.g., correcting the model's perception of a lesion's shape). The SVCT can then update its final diagnosis based on this expert feedback, enabling a true collaborative diagnostic environment between AI and medical professionals.

Conclusion and Future Work

The Stable Vision Concept Transformer proves that interpretability does not have to come at the cost of performance. By fusing concept-based reasoning with robust diffusion-based smoothing, the SVCT provides a reliable framework for medical AI. While current research focuses on Gaussian noise, future work will explore the model's resilience against other clinical artifacts, such as motion blur or sensor-specific distortions, to further prepare the system for real-world hospital environments.

Summary updated

BibTeX
@inproceedings{hu2025stable,
  author = {Lijie Hu and Songning Lai and Yuan Hua and Shu Yang and Jingfeng Zhang and Di Wang},
  title = {{Stable Vision Concept Transformers for Medical Diagnosis}},
  booktitle = {ECML/PKDD},
  year = {2025},
  url = {https://openreview.net/forum?id=4VfqT2gwrH},
}
Figure