From Linear Probing to Joint-Weighted Token Hierarchy: A Foundation Model Bridging Global and Cellular Representations in Biomarker Detection

Jingsong Liu, Han Li, Nassir Navab, Peter J. SchüfflerarXiv 2025 · DOI · arXiv

Abstract

Pathology Foundation Models (PFMs) typically utilize global "Class Tokens" for downstream tasks, often discarding "Local Tokens" that contain essential cellular-level information. We present Joint-Weighted Token Hierarchy (JWTH), a model that bridges this gap using a novel attention-pooling mechanism and Gram-anchoring. Pretrained on 84 million tissue patches, JWTH achieves state-of-the-art performance in biomarker detection, demonstrating significant improvements over existing models like UNI and Virchow2.

Introduction

In digital pathology, Vision Transformers (ViTs) have revolutionized tissue analysis. However, a significant limitation persists: standard "linear probing" techniques focus exclusively on the global Class Token, effectively ignoring the local tokens that represent sub-regional morphology. In clinical diagnostics, these fine-grained details—such as nuclear shape and cell-to-cell interactions—are often critical. JWTH addresses this by fusing global context with local cellular representations.

Methodology: Rescuing Local Tokens

The JWTH framework introduces two key innovations to preserve and utilize local information:

  1. Gram-Anchoring Post-Training: To prevent the "collapse" of local token embeddings during large-scale self-supervised learning, we employ Gram-anchoring. This technique ensures diversity among local tokens, capturing a broad spectrum of cellular features.
  2. Attention-Based Hierarchical Pooling: Instead of relying solely on the Class Token, JWTH uses an attention mechanism where the Class Token acts as a query to weight the importance of local tokens. This creates a unified, hierarchical representation of the tissue.
Comparison of standard linear probing versus the JWTH hierarchical approach.

Experimental Results

JWTH was pretrained on a massive dataset of 84 million patches from over 84,000 whole-slide images. We evaluated the model across six diverse tasks, including colorectal cancer detection (NCT-CRC-HE-100K), polyp pattern detection (MHIST), and breast cancer histology (BACH).

The results indicate that JWTH consistently outperforms leading PFMs. As shown in the radar chart below, JWTH maintains a superior performance profile across multiple benchmarks compared to models like UNI, UNI2, and DinoV3.

Radar plot comparing JWTH performance against state-of-the-art models across various pathology benchmarks.

Key findings include:

  • Superior Accuracy: JWTH achieved up to an 8.3% increase in balanced accuracy (BACC) over the next-best model in specific tasks.
  • Robustness: The integration of random staining augmentations during pretraining significantly improved the model's resilience to "stain shift" across different laboratory environments.
  • Ablation Insights: Our studies confirmed that attention pooling alone contributes a 3.3% performance boost, validating the importance of hierarchical feature extraction.

Conclusion

JWTH represents a significant step forward in computational pathology by demonstrating that local cellular features are not merely noise but vital components of a robust diagnostic model. By bridging the gap between global and local representations, JWTH provides a more comprehensive and clinically relevant foundation for biomarker detection. Future work will explore scaling this hierarchical approach and integrating multi-modal data for enhanced clinical utility.

Summary updated

BibTeX
@misc{liu2025from,
  author = {Jingsong Liu and Han Li and Nassir Navab and Peter J. Schüffler},
  title = {{From Linear Probing to Joint-Weighted Token Hierarchy: A Foundation Model Bridging Global and Cellular Representations in Biomarker Detection}},
  year = {2025},
  eprint = {2511.05150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.05150},
}
Figure