Beyond Linearity: Squeeze-and-Recalibrate Blocks for Few-Shot Whole Slide Image Classification

Conghao Xiong, Zhengrui Guo, Zhe Xu, Yifei Zhang, Raymond Kai-Yu Tong, Si Yong Yeo, Hao Chen, Joseph J. Y. Sung, Irwin King

Introduction

Whole Slide Images (WSIs)—gigapixel-resolution digital scans of pathology slides—are the gold standard for disease diagnosis. However, training AI to analyze them is often hindered by data scarcity. Multiple Instance Learning (MIL) is the standard paradigm, treating each slide as a "bag" of thousands of small image patches. In few-shot learning scenarios, where only a few labeled slides are available, models almost inevitably overfit, memorizing noise instead of learning underlying biological features.

In their recent work, researchers argue that overfitting is not merely a consequence of small sample sizes but is driven by how models "shatter" the geometric structure of the data. They demonstrate that standard linear layers in pathology models distort the low-dimensional manifold of features extracted from foundation models, leading to poor generalization.

The Challenge: Manifold Distortion

Modern pathology AI relies on Pathology Foundation Models (PFMs) to extract meaningful features. Spectral analysis of these features reveals a critical insight: although they exist in high-dimensional space (e.g., 512 dimensions for CONCH), they reside on a low-dimensional manifold with an effective rank of approximately 30.

Standard MIL models utilize linear layers that are "geometry-agnostic." These layers act destructively, twisting and warping the fragile manifold. Using tangent space analysis, the authors visually demonstrated that trained linear layers significantly increase "geometric drift," essentially warping the data's natural curvature. In few-shot settings, this distortion makes it impossible for the model to generalize effectively.

Analysis of feature geometry: spectral decay, t-SNE visualizations, and tangent space drift.

The Manifold Residual (MR) Block

To preserve this geometric integrity, the authors propose the Manifold Residual (MR) block, a plug-and-play module designed to replace standard linear layers. The MR block decouples feature mapping into two parallel pathways:

  1. The Geometric Anchor: A fixed, frozen random matrix initialized with a Kaiming uniform distribution. Based on random projection theory (such as the Johnson-Lindenstrauss Lemma), this path approximately preserves distances and the local topology of the data. It acts as a stabilizer, ensuring the manifold does not collapse during training.
  2. The Low-rank Residual Path (LRP): A trainable path consisting of two small matrices with a bottleneck of rank $r$. This path is designed to learn only the task-specific residuals—the small adjustments needed for the specific medical task.

By summing these two paths, the model gains the flexibility to learn new tasks without discarding the rich geometric priors learned by the foundation model during pre-training.

Architecture of the Manifold Residual (MR) block and its integration into the MIL pipeline.

Experimental Results

The researchers validated their approach across several diverse benchmarks, including Camelyon16 (breast cancer metastasis), TCGA-NSCLC (lung cancer subtyping), and TCGA-RCC (renal cell carcinoma).

  • State-of-the-Art Performance: The MR block consistently outperformed standard MIL models (like ABMIL, TransMIL, and CATE), especially in extreme few-shot settings (e.g., 2 or 4 slides per class). On Camelyon16 (8-shot), the MR-CATE model achieved an AUC of 88.0%, compared to the baseline CATE's 79.8%.
  • Efficiency: MR-enhanced models achieved these results while using significantly fewer trainable parameters—often a 50-60% reduction compared to standard architectures.
  • Improved Interpretability: Heatmaps generated by MR-blocks were cleaner and more accurately delineated tumor boundaries. Standard models often produced noisy or "shattered" attention maps, whereas the MR block maintained spatial coherence.

Conclusion

The MR block represents a shift in how we approach few-shot learning in medical imaging. By prioritizing the preservation of the data's intrinsic manifold, it enables models to remain robust even with minimal supervision. Future work may explore data-driven anchors, such as PCA projections, to further incorporate dataset-specific information without increasing the risk of overfitting.

Summary updated

BibTeX
@inproceedings{xiong2026beyond,
  author = {Conghao Xiong and Zhengrui Guo and Zhe Xu and Yifei Zhang and Raymond Kai-Yu Tong and Si Yong Yeo and Hao Chen and Joseph J. Y. Sung and Irwin King},
  title = {{Beyond Linearity: Squeeze-and-Recalibrate Blocks for Few-Shot Whole Slide Image Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=HBP9uSEYME},
}
Figure