ParseCaps: An Interpretable Parsing Capsule Network for Medical Image Diagnosis

Xinyu Geng, Jiaming Wang, Xiaolin Huang, Fanglin Chen, Jun Xu

Abstract

Deep learning has excelled in medical image classification, but its clinical application is limited by poor interpretability. Capsule networks, known for encoding hierarchical relationships and spatial features, show potential in addressing this issue. Nevertheless, traditional capsule networks often underperform due to their shallow structures, and deeper variants lack hierarchical architectures, thereby compromising interpretability. This paper introduces a novel capsule network, ParseCaps, which utilizes the sparse axial attention routing and parse convolutional capsule layer to form a parse-tree-like structure, enhancing both depth and interpretability. Firstly, sparse axial attention routing optimizes connections between child and parent capsules, as well as emphasizes the weight distribution across instantiation parameters of parent capsules. Secondly, the parse convolutional capsule layer generates capsule predictions aligning with the parse tree. Finally, based on the loss design that is effective whether concept ground truth exists or not, ParseCaps advances interpretability by associating each dimension of the global capsule with a comprehensible concept, thereby facilitating clinician trust and understanding of the model's classification results. Experimental results on three medical datasets show that ParseCaps not only outperforms other capsule network variants in classification accuracy and robustness, but also provides interpretable explanations, regardless of the availability of concept labels.

Introduction

Deep learning has revolutionized medical imaging, yet it faces a significant "black box" problem. While a model might accurately identify a pathology, it often fails to explain the reasoning behind its decision in a manner that clinicians can verify. This lack of transparency remains a major barrier to the integration of AI in clinical workflows.

Capsule Networks (CapsNets) were proposed to address these limitations by using vectors (capsules) instead of scalar neurons to represent entities and their spatial relationships. However, traditional CapsNets are typically shallow and struggle with complex medical data. Attempts to deepen these networks often result in the loss of the "parse tree" structure—the intuitive hierarchy where low-level features (like edges) combine to form sub-components (like lesion borders), which eventually constitute a whole entity. ParseCaps aims to restore this hierarchy, creating a deep network that is both high-performing and inherently explainable.

The ParseCaps Architecture

To create a deep, interpretable hierarchy, ParseCaps introduces three primary structural innovations: the Parse Convolutional Capsule (PConvCaps) layer, Sparse Axial Attention (SAA) routing, and a concept-linked loss framework.

The ParseCaps architecture, featuring a hierarchical block-based structure with PConvCaps layers and SAA routing.

Parse Convolutional Capsule (PConvCaps) Layer

Standard convolutional layers often increase feature redundancy as they deepen. In contrast, the PConvCaps layer reduces the number of capsules while increasing their dimensionality as the network progresses. This design mimics a natural parse tree, narrowing down from numerous fine-grained details to a single "global capsule" that represents the entire image.

Sparse Axial Attention (SAA) Routing

Routing determines how information is passed from "child" capsules to "parent" capsules. ParseCaps utilizes SAA routing to enforce clarity:

  • Sparse Attention: By employing an $\alpha$-entmax function, the model zeros out weak connections, forcing a more distinct, tree-like structure.
  • Axial Attention: Rather than treating a capsule as a single block of data, SAA examines individual dimensions (instantiation parameters). This allows the model to learn which specific features—such as color, texture, or shape—are most critical for a specific diagnosis.

Achieving Interpretability

The core strength of ParseCaps lies in its ability to map complex image features to human-understandable clinical concepts. By funneling hierarchical information into a final global capsule, the architecture allows researchers to assign specific dimensions of that vector to clinical markers.

Overview of the ParseCaps interpretability framework, mapping concept capsules to a global capsule through a parse-tree structure.

This mapping is supported by a specialized loss framework that functions with or without expert-labeled concepts. Using Triplet Loss, the model aligns specific image regions with clinical terms (e.g., "atypical pigment network") in a shared latent space. This transforms the final classification into a verifiable checklist of visible features, fostering clinician trust.

Experimental Evaluation

ParseCaps was evaluated across several benchmark datasets, including CE-MRI (brain tumors), PH² and Derm7pt (skin lesions), and robustness tests on MNIST/affNIST.

  • State-of-the-Art Accuracy: ParseCaps achieved a remarkable 99.38% accuracy on the CE-MRI dataset, outperforming established capsule variants like DeepCaps and OrthCaps. On the PH² skin dataset, it reached 97.53% accuracy.
  • Robustness to Viewpoint: Due to its preservation of spatial hierarchies, ParseCaps demonstrated superior performance over standard CNNs when recognizing objects subjected to rotation or scaling.
  • Explanation Fidelity: In ablation studies, the combination of Triplet and Presentation losses reduced "Explanation Error"—a metric measuring alignment with ground-truth concepts—from 4.02 to 0.98. Qualitative visualizations confirmed that the model correctly highlights clinical markers used by dermatologists, such as "Blue-Whitish Veil."

Conclusion and Future Work

ParseCaps represents a significant advancement in "glass-box" medical AI. By enforcing a sparse, hierarchical structure, it achieves high diagnostic accuracy while providing clear, concept-based explanations.

Despite these gains, challenges remain. The current architecture is "parse-tree-like" rather than a strict tree; a stricter hierarchy might further improve interpretability but could potentially impact raw performance. Additionally, mapping discovered prototypes to medical concepts still requires expert input. Future research will focus on enhancing unsupervised interpretability to reduce the reliance on expensive expert annotations, further scaling the model's utility across diverse medical domains.

Summary updated

Abstract

Deep learning has excelled in medical image classification, but its clinical application is limited by poor interpretability. Capsule networks, known for encoding hierarchical relationships and spatial features, show potential in addressing this issue. Nevertheless, traditional capsule networks often underperform due to their shallow structures, and deeper variants lack hierarchical architectures, thereby compromising interpretability. This paper introduces a novel capsule network, ParseCaps, which utilizes the sparse axial attention routing and parse convolutional capsule layer to form a parse-tree-like structure, enhancing both depth and interpretability. Firstly, sparse axial attention routing optimizes connections between child and parent capsules, as well as emphasizes the weight distribution across instantiation parameters of parent capsules. Secondly, the parse convolutional capsule layer generates capsule predictions aligning with the parse tree. Finally, based on the loss design that is effective whether concept ground truth exists or not, ParseCaps advances interpretability by associating each dimension of the global capsule with a comprehensible concept, thereby facilitating clinician trust and understanding of the model's classification results. Experimental results on three medical datasets show that ParseCaps not only outperforms other capsule network variants in classification accuracy and robustness, but also provides interpretable explanations, regardless of the availability of concept labels.
BibTeX
@inproceedings{geng2025parsecaps,
  author = {Xinyu Geng and Jiaming Wang and Xiaolin Huang and Fanglin Chen and Jun Xu},
  title = {{ParseCaps: An Interpretable Parsing Capsule Network for Medical Image Diagnosis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=7xpYNj5GSJ},
}
Figure