Abstract
These results demonstrate that PAT-MIL effectively mitigates staining variability and enhances cross-center generalization through the collaborative modeling of visual and textual modalities. It provides a robust solution for colorectal lesion classification without relying on pixel-level annotations, advancing the field of multimodal pathological image analysis.
Introduction
In digital pathology, diagnosing colorectal lesions from Whole Slide Images (WSIs) presents a significant challenge. These gigapixel-scale images often contain diagnostic features in only a minute fraction of the tissue, creating a "needle in a haystack" scenario. Due to the high cost of manual pixel-level annotation, researchers typically employ Weakly Supervised Learning, specifically Multiple Instance Learning (MIL), where labels are assigned to entire slides rather than individual pixels.
However, standard MIL models face two primary obstacles:
- The Semantic Gap: Most models rely exclusively on visual patterns, neglecting the structured expert knowledge and textual descriptions pathologists use for diagnosis.
- Domain Shift: Variability in tissue staining across different medical centers and scanners often leads to performance degradation when models encounter unfamiliar color distributions.
To address these issues, this paper introduces PAT-MIL (Pathology-Attention-MIL), a framework designed to integrate visual evidence with semantic guidance from expert-defined pathological text.
Methodology
The PAT-MIL framework enhances the standard MIL pipeline through three innovative components:
- Expert-Defined Text Prototypes: Pathologists curated detailed text descriptions for five lesion categories, serving as semantic anchors for the model.
- Dynamic Prototype Refinement: The model iteratively adjusts these category centers based on the distribution of visual features in the slides, ensuring the semantic definitions align with the actual data.
- Dual-Loss Dynamic Weighting: A real-time optimization strategy balances visual clustering and text-alignment losses based on their gradient magnitudes, focusing the model's learning where it is most needed.
The system utilizes the Virchow foundation model as its primary feature extractor to capture intricate morphological details.
Dataset and Experimental Setup
The study leveraged a comprehensive collection of 5,062 H&E stained WSIs across several cohorts to ensure robust validation:
- Internal-XJ: 1,756 slides used for training and internal validation.
- External-LZ (Xijing Hospital): 1,756 slides used as an independent external test set to prove generalization.
- Public Datasets: Including UniToPatho (292 slides) and IMP-CRS 2024 (2,032 slides).
Results
PAT-MIL consistently outperformed established baselines such as ABMIL, CLAM, and DSMIL across multiple benchmarks:
- Internal Accuracy: The model achieved 86.45% accuracy (AUC = 0.9624), a 2.19% improvement over the best baseline (DSMIL).
- External Generalization: On the UniToPatho dataset, PAT-MIL reached 84.09% accuracy, representing a significant 5.68% increase over competing models.
- Clinical Precision: In distinguishing cancerous from non-cancerous cases on the external test set, the model demonstrated 99.6% specificity and 99.0% precision.
- Ablation Studies: Removing the text alignment module led to a marked decrease in performance, confirming that multimodal integration is critical to the framework's success.
Discussion
A key insight from this research is the role of text as a regularizer. While visual features like color and texture vary with staining protocols, pathological descriptions (e.g., "irregular glandular structures") remain invariant. By aligning image features with these descriptions, PAT-MIL achieves superior robustness against domain shift. Furthermore, the text-guided attention mechanism improves interpretability, allowing the model to highlight regions of clinical interest that correspond to expert descriptions.
Conclusion
PAT-MIL represents a significant advancement in multimodal pathological image analysis. By bridging the gap between visual data and expert knowledge, it provides a robust and interpretable solution for colorectal lesion classification. Future work will explore multi-scale feature extraction and adaptive text generation to further enhance the model's flexibility and clinical utility across diverse medical institutions.
Summary updated