End-to-end prediction of clinical outcomes in head and neck squamous cell carcinoma with foundation model-based multiple instance learning

Asier Rabasco Meneghetti, Marta Ligero, Johannes Kuehn, Steffen Löck, Zunamys I. Carrero, Raquel Pérez-López, Keno K. Bressem, Titus J. Brinker, Alexander T. Pearson, Daniel Truhn, Sven Nebelung, Jakob Nikolas KathermedRxiv 2025 · DOI

Abstract

Foundation model (FM)-based multiple instance learning (MIL) demonstrates significant promise in predicting clinical outcomes for head and neck squamous cell carcinoma (HNSCC). This study shows that end-to-end MIL models achieve performance comparable or superior to traditional handcrafted radiomics, particularly enhancing prognostic accuracy for non-HPV+ patients.

Introduction

Predicting treatment response in head and neck squamous cell carcinoma (HNSCC) remains a clinical challenge, with 5-year survival rates varying widely between 30% and 70%. Current prognostic tools, such as TNM staging and HPV status, often fail to capture the intricate biological heterogeneity of these tumors. While traditional radiomics—the extraction of high-dimensional mathematical features from medical imaging—has been proposed as a solution, it is often hampered by the need for manual segmentation, lack of standardization, and labor-intensive feature engineering.

Deep learning offers an alternative, yet its application is frequently limited by the scarcity of large, annotated medical datasets. This research investigates the utility of foundation models—AI architectures pre-trained on massive, diverse datasets—integrated into a Multiple Instance Learning (MIL) framework to extract prognostic biomarkers directly from routine CT scans.

Methods

The study employed a Multiple Instance Learning (MIL) framework to process 3D CT data without requiring slice-by-slice annotations. In this "bag-of-instances" approach, the entire set of tumor-containing slices is treated as a single entity for patient-level prediction.

The workflow consists of three primary stages:

  1. Feature Extraction: Pre-trained foundation models (e.g., BioMedClip, ResNet50) serve as backbones to generate feature embeddings from CT slices.
  2. Aggregation: A MIL Transformer processes these embeddings to learn spatial and contextual relationships between slices, outputting a unified risk score.
  3. Endpoint Prediction: The models were trained to predict three critical clinical outcomes: 2-year overall survival (OS), locoregional control (LRC), and freedom from distant metastasis (FFDM).
Overview of the study design (A), MIL Transformer architecture (B), and the 2D (C) and 3D (D) feature extraction workflows.

Datasets and Patient Selection

The researchers evaluated their approach using a large-scale dataset comprising 2,485 patients across three distinct cohorts. The primary cohort, RADCURE (Toronto, Canada), was utilized for training (n=1,464) and internal testing (n=606). External validation was performed on the HN1 cohort (Maastricht, Netherlands; n=131) and the HN-PET-CT cohort (Canada; n=284).

Flowchart detailing the patient selection process, including inclusion and exclusion criteria across the RADCURE, HN1, and HN-PET-CT cohorts.

Results and Discussion

The end-to-end MIL approach demonstrated robust predictive performance, often surpassing traditional methods. Key findings include:

  • Dimensionality and Performance: 2D MIL models utilizing axial slices surprisingly outperformed more complex 3D or multiview architectures. This suggests that prognostic information in HNSCC is highly concentrated in axial cross-sections and that current 3D foundation models may require more specialized pre-training for head and neck anatomy.
  • Comparison with Radiomics: The MIL models achieved 2-year test AUROCs of 0.75–0.84 for overall survival. Notably, in the HN-PET-CT cohort, the MIL model (AUROC 0.84) significantly outperformed traditional handcrafted radiomics (AUROC 0.68, p=0.012).
  • Clinical Utility in non-HPV+ Subgroups: While HPV status is a strong prognostic indicator, the MIL scores provided the most substantial improvement to clinical baseline models in non-HPV+ patients. For this high-risk group, combining clinical data with MIL features increased the AUROC for OS and FFDM to 0.87.
  • Risk Stratification: The models effectively stratified patients into high- and low-risk categories, yielding Hazard Ratios (HR) between 2.14 and 4.77.

Conclusion

This study validates the potential of foundation model-based MIL for automated, end-to-end clinical outcome prediction in HNSCC. By matching or exceeding the performance of labor-intensive radiomics, this approach simplifies clinical workflows and provides critical prognostic insights, particularly for non-HPV+ patients. Future research should focus on developing 3D foundation models specifically tailored for oncology and exploring the prognostic value of the peritumoral environment.

Summary updated

Abstract

FM-based MIL demonstrates promise in HNSCC risk prediction, showing similar or superior performance to radiomics and enhancing clinical baselines in non-HPV+ patients.
BibTeX
@misc{meneghetti2025end,
  author = {Asier Rabasco Meneghetti and Marta Ligero and Johannes Kuehn and Steffen Löck and Zunamys I. Carrero and Raquel Pérez-López and Keno K. Bressem and Titus J. Brinker and Alexander T. Pearson and Daniel Truhn and Sven Nebelung and Jakob Nikolas Kather},
  title = {{End-to-end prediction of clinical outcomes in head and neck squamous cell carcinoma with foundation model-based multiple instance learning}},
  year = {2025},
  doi = {10.1101/2025.01.22.25320517},
  howpublished = {medRxiv},
}
Figure