Co-HSF: Resource-Efficient One-Shot Semi-Supervised Adaptation of Histopathology Foundation Models

Luca Cerny Oliveira, Kartik Patwari, Xiaoguang Zhu, Sen-Ching S. Cheung, Brittany N. Dugger, Chen-Nee Chuah

Introduction

Automating the analysis of biopsy slides through histopathology is a critical advancement for modern medicine. However, the field faces a significant "data bottleneck." Training robust deep learning models typically necessitates thousands of images meticulously labeled by expert pathologists—a process that is prohibitively expensive, time-consuming, and fraught with privacy concerns.

While foundation models like CONCH offer zero-shot capabilities, their performance in specialized medical domains often falls short of clinical requirements. Fine-tuning these models with minimal data (e.g., one-shot learning) frequently results in overfitting, where the model memorizes specific examples rather than learning generalizable features. Furthermore, many existing few-shot methods are computationally intensive, hindering deployment on standard hospital hardware.

Methodology

To address these challenges, we present Co-HSF (Co-filtered Histopathology Semi-Supervised Few-Shot), a framework designed to adapt large histopathology foundation models using only one labeled example per class.

The Co-HSF framework utilizes a dual-model training loop and co-filtering to leverage unlabeled data effectively.

The architecture is built upon three core components:

  1. Dual-SSFSL Training Loop: The system employs a "Teacher" and a "Student" model, both utilizing the CONCH vision encoder, a state-of-the-art model pre-trained specifically on pathology data.
  2. Co-filtering (CF) Pseudo-labeling: This mechanism allows the model to utilize vast amounts of unlabeled clinical data. A pseudo-label is only assigned to an unlabeled image if both the Teacher and Student models reach a consensus across both the original and augmented versions of the image. This "double-check" strategy effectively filters out the noisy labels that often degrade semi-supervised learning performance.
  3. Inductive Inference: Unlike transductive methods that require access to unlabeled data during prediction, Co-HSF trains a standalone classifier head. This approach bypasses the need for text encoding at test time, significantly reducing computational overhead.
The Co-filtering process ensures pseudo-label reliability by requiring consistency across multiple model views and data augmentations.

Experimental Results

The framework was evaluated on two major benchmarks: NCT-CRC-HE (100,000 colorectal cancer patches) and PatchCamelyon (327,000 breast cancer patches).

  • Performance Gains: On the 9-class NCT dataset, Co-HSF achieved 90.1% accuracy with just one labeled sample per class, a dramatic improvement over the 51.7% accuracy of the zero-shot CONCH baseline.
  • Efficiency: Co-HSF demonstrated a 53% increase in inference speed compared to zero-shot approaches.
  • Resource Optimization: The model's memory footprint was reduced to 952 MiB of VRAM, significantly lower than the 2200+ MiB required by standard foundation model adaptation methods. This efficiency facilitates deployment on modest clinical hardware.
  • Robustness: Co-HSF showed lower performance variability across different random seeds, indicating high stability regardless of the specific one-shot sample selected.

Discussion and Conclusion

Co-HSF demonstrates that model consistency is a powerful proxy for ground truth in low-data regimes. By enforcing agreement across different model architectures and data augmentations, the framework successfully leverages unlabeled data without propagating errors. Furthermore, the transition from vision-language comparison to direct visual classification streamlines the inference pipeline, making high-performance histopathology analysis accessible in resource-constrained environments. Future work will focus on addressing class imbalance and evaluating the model's performance across diverse, multi-center clinical datasets.

Summary updated

Abstract

Automated analysis of histopathological samples has greatly augmented the ability of experts to perform deep phenotyping on biological samples. Current state-of-the-art (SOTA) methods for histopathology image classification rely on training deep neural networks with large annotated datasets, which can be costly to obtain. Recent studies propose to bypass annotated datasets by leveraging pre-trained foundation models (e.g. visual-language models) for zero-shot predictions. Moreover, fine-tuning these models enhances performance while requiring minimal labeled data (e.g. one-shot fine-tuning). However, one-shot fine-tuned performance of histopathology foundation models on image classification tasks is understudied. In this work, we first explore the use of semi-supervised few-shot learning (SSFSL) for fine-tuning histopathology foundation models on one-shot datasets with unlabeled samples. We find SOTA SSFSL methods improve fine-tuning performance, but their pseudo-labeling (i.e. assigning labels to unlabeled samples) strategies can increase inference times over zero-shot. We then propose a Co-filtered Histopathology Semi-Supervised Few-Shot (Co-HSF) pipeline: a dual-SSFSL (i.e. with teacher and student models) training loop followed by a co-filtering (CF) pseudo-labeling strategy to efficiently leverage unlabeled data for improved semi-supervised performance and reduced inference times. Using the National Center for Tumor Disease Colorectal Cancer Dataset (NCT-CRC-HE), we show our proposed module achieves 38.4% improvement in accuracy over zero-shot performance with only 9 labeled samples and over 53% faster inference times, while also outperforming other fine-tuning and SSFSL methods.
BibTeX
@inproceedings{oliveira2025co,
  author = {Luca Cerny Oliveira and Kartik Patwari and Xiaoguang Zhu and Sen-Ching S. Cheung and Brittany N. Dugger and Chen-Nee Chuah},
  title = {{Co-HSF: Resource-Efficient One-Shot Semi-Supervised Adaptation of Histopathology Foundation Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaaiss.v5i1.35545},
}
Figure