Abstract
Pathology Foundation Models (PFMs) typically utilize global "Class Tokens" for downstream tasks, often discarding "Local Tokens" that contain essential cellular-level information. We present Joint-Weighted Token Hierarchy (JWTH), a model that bridges this gap using a novel attention-pooling mechanism and Gram-anchoring. Pretrained on 84 million tissue patches, JWTH achieves state-of-the-art performance in biomarker detection, demonstrating significant improvements over existing models like UNI and Virchow2.
Introduction
In digital pathology, Vision Transformers (ViTs) have revolutionized tissue analysis. However, a significant limitation persists: standard "linear probing" techniques focus exclusively on the global Class Token, effectively ignoring the local tokens that represent sub-regional morphology. In clinical diagnostics, these fine-grained details—such as nuclear shape and cell-to-cell interactions—are often critical. JWTH addresses this by fusing global context with local cellular representations.
Methodology: Rescuing Local Tokens
The JWTH framework introduces two key innovations to preserve and utilize local information:
- Gram-Anchoring Post-Training: To prevent the "collapse" of local token embeddings during large-scale self-supervised learning, we employ Gram-anchoring. This technique ensures diversity among local tokens, capturing a broad spectrum of cellular features.
- Attention-Based Hierarchical Pooling: Instead of relying solely on the Class Token, JWTH uses an attention mechanism where the Class Token acts as a query to weight the importance of local tokens. This creates a unified, hierarchical representation of the tissue.
Experimental Results
JWTH was pretrained on a massive dataset of 84 million patches from over 84,000 whole-slide images. We evaluated the model across six diverse tasks, including colorectal cancer detection (NCT-CRC-HE-100K), polyp pattern detection (MHIST), and breast cancer histology (BACH).
The results indicate that JWTH consistently outperforms leading PFMs. As shown in the radar chart below, JWTH maintains a superior performance profile across multiple benchmarks compared to models like UNI, UNI2, and DinoV3.
Key findings include:
- Superior Accuracy: JWTH achieved up to an 8.3% increase in balanced accuracy (BACC) over the next-best model in specific tasks.
- Robustness: The integration of random staining augmentations during pretraining significantly improved the model's resilience to "stain shift" across different laboratory environments.
- Ablation Insights: Our studies confirmed that attention pooling alone contributes a 3.3% performance boost, validating the importance of hierarchical feature extraction.
Conclusion
JWTH represents a significant step forward in computational pathology by demonstrating that local cellular features are not merely noise but vital components of a robust diagnostic model. By bridging the gap between global and local representations, JWTH provides a more comprehensive and clinically relevant foundation for biomarker detection. Future work will explore scaling this hierarchical approach and integrating multi-modal data for enhanced clinical utility.
Summary updated