A pathology-attention multi-instance learning framework for multimodal classification of colorectal lesions

Freddie H. Fu, Xuemei Zhang, Zhaoxuan Wang, Lin Xie, Mingye Fu, Jing Peng, Jianfeng Wu, Zhe Wang, Tian Guan, Yonghong He, Jin‐Shun Lin, Lianghui Zhu, Wenbin DaiFrontiers in Pharmacology 2025 · DOI

Abstract

These results demonstrate that PAT-MIL effectively mitigates staining variability and enhances cross-center generalization through the collaborative modeling of visual and textual modalities. It provides a robust solution for colorectal lesion classification without relying on pixel-level annotations, advancing the field of multimodal pathological image analysis.

Introduction

In digital pathology, diagnosing colorectal lesions from Whole Slide Images (WSIs) presents a significant challenge. These gigapixel-scale images often contain diagnostic features in only a minute fraction of the tissue, creating a "needle in a haystack" scenario. Due to the high cost of manual pixel-level annotation, researchers typically employ Weakly Supervised Learning, specifically Multiple Instance Learning (MIL), where labels are assigned to entire slides rather than individual pixels.

However, standard MIL models face two primary obstacles:

  1. The Semantic Gap: Most models rely exclusively on visual patterns, neglecting the structured expert knowledge and textual descriptions pathologists use for diagnosis.
  2. Domain Shift: Variability in tissue staining across different medical centers and scanners often leads to performance degradation when models encounter unfamiliar color distributions.

To address these issues, this paper introduces PAT-MIL (Pathology-Attention-MIL), a framework designed to integrate visual evidence with semantic guidance from expert-defined pathological text.

Methodology

The PAT-MIL framework enhances the standard MIL pipeline through three innovative components:

  • Expert-Defined Text Prototypes: Pathologists curated detailed text descriptions for five lesion categories, serving as semantic anchors for the model.
  • Dynamic Prototype Refinement: The model iteratively adjusts these category centers based on the distribution of visual features in the slides, ensuring the semantic definitions align with the actual data.
  • Dual-Loss Dynamic Weighting: A real-time optimization strategy balances visual clustering and text-alignment losses based on their gradient magnitudes, focusing the model's learning where it is most needed.

The system utilizes the Virchow foundation model as its primary feature extractor to capture intricate morphological details.

Dataset and Experimental Setup

The study leveraged a comprehensive collection of 5,062 H&E stained WSIs across several cohorts to ensure robust validation:

  • Internal-XJ: 1,756 slides used for training and internal validation.
  • External-LZ (Xijing Hospital): 1,756 slides used as an independent external test set to prove generalization.
  • Public Datasets: Including UniToPatho (292 slides) and IMP-CRS 2024 (2,032 slides).
Overview of the PAT-MIL framework, dataset distribution, and architectural components.

Results

PAT-MIL consistently outperformed established baselines such as ABMIL, CLAM, and DSMIL across multiple benchmarks:

  • Internal Accuracy: The model achieved 86.45% accuracy (AUC = 0.9624), a 2.19% improvement over the best baseline (DSMIL).
  • External Generalization: On the UniToPatho dataset, PAT-MIL reached 84.09% accuracy, representing a significant 5.68% increase over competing models.
  • Clinical Precision: In distinguishing cancerous from non-cancerous cases on the external test set, the model demonstrated 99.6% specificity and 99.0% precision.
  • Ablation Studies: Removing the text alignment module led to a marked decrease in performance, confirming that multimodal integration is critical to the framework's success.

Discussion

A key insight from this research is the role of text as a regularizer. While visual features like color and texture vary with staining protocols, pathological descriptions (e.g., "irregular glandular structures") remain invariant. By aligning image features with these descriptions, PAT-MIL achieves superior robustness against domain shift. Furthermore, the text-guided attention mechanism improves interpretability, allowing the model to highlight regions of clinical interest that correspond to expert descriptions.

Conclusion

PAT-MIL represents a significant advancement in multimodal pathological image analysis. By bridging the gap between visual data and expert knowledge, it provides a robust and interpretable solution for colorectal lesion classification. Future work will explore multi-scale feature extraction and adaptive text generation to further enhance the model's flexibility and clinical utility across diverse medical institutions.

Summary updated

Abstract

These results demonstrate that PAT-MIL effectively mitigates staining variability and enhances cross-center generalization through the collaborative modeling of visual and textual modalities. It provides a robust solution for colorectal lesion classification without relying on pixel-level annotations, advancing the field of multimodal pathological image analysis.
BibTeX
@inproceedings{fu2025pathology,
  author = {Freddie H. Fu and Xuemei Zhang and Zhaoxuan Wang and Lin Xie and Mingye Fu and Jing Peng and Jianfeng Wu and Zhe Wang and Tian Guan and Yonghong He and Jin‐Shun Lin and Lianghui Zhu and Wenbin Dai},
  title = {{A pathology-attention multi-instance learning framework for multimodal classification of colorectal lesions}},
  booktitle = {Frontiers in Pharmacology},
  year = {2025},
  doi = {10.3389/fphar.2025.1592950},
}
Figure