Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study

Sophia J. Wagner, Daniel Reisenbüchler, Nicholas P. West, Jan Niehues, Jiefu Zhu, Sebastian Foersch, Gregory Patrick Veldhuizen, Philip Quirke, Heike I. Grabsch, Piet A. van den Brandt, Gordon Hutchins, Susan D. Richman, Tanwei Yuan, Rupert Langer, Josien C.a. Jenniskens, Kelly Offermans, Wolfram Mueller, Richard Gray, Stephen B. Gruber, Joel K. Greenson, Gad Rennert, Joseph D. Bonner, Daniel Schmolze, Jitendra Jonnagaddala, Nicholas J. Hawkins, Robyn L. Ward, Dion Morton, Michel Seymour, Laura Magill, Marta Nowak, Jennifer Hay, Viktor H. Koelzer, David N. Church, Christian Matek, Carol Geppert, Chaolong Peng, Cheng Zhi, Xiaoming Ouyang, Jacqueline A. James, Maurice B. Loughrey, Manuel Salto‐Tellez, Hermann Brenner, Michael Hoffmeister, Daniel Truhn, Julia A. Schnabel, Melanie Boxberg, Tingying Peng, Jakob Nikolas Kather, Enric Domingo, Joanne Edwards, Bengt Glimelius, Ismail Gögenür, Andrea Harkin, Timothy Iveson, Emma Jaeger, Caroline Kelly, Rachel Kerr, Noori Maka, Hannah Morgan, Karin A. Oien, Clare Orange, Claire Palles, Campbell S.d. Roxburgh, Owen J. Sansom, Mark Saunders, Ian TomlinsonCancer 2023 · DOI

Abstract

Precision oncology for colorectal cancer (CRC) is currently limited by the time and cost associated with molecular assays for biomarkers like Microsatellite Instability (MSI). While deep learning has shown potential in predicting these markers from H&E pathology slides, traditional convolutional neural networks (CNNs) often struggle with global context and generalization across different clinical settings. This study introduces a fully transformer-based pipeline trained on a massive multicentric dataset of over 9,000 patients. The model achieves state-of-the-art performance, demonstrating high sensitivity and, crucially, clinical-grade accuracy on endoscopic biopsies even when trained solely on surgical resections. These findings suggest a path toward immediate, automated biomarker screening at the point of initial diagnosis.

Introduction

The identification of genetic biomarkers such as Microsatellite Instability (MSI) and mutations in BRAF or KRAS is essential for tailoring treatment in colorectal cancer. However, the current gold standard—molecular testing via PCR or next-generation sequencing—can take weeks to process. This delay can postpone the initiation of targeted therapies or immunotherapy.

Digital pathology offers a solution by using deep learning (DL) to predict molecular status directly from routine hematoxylin and eosin (H&E) stained slides. Despite this potential, existing DL methods face significant hurdles: they often fail to capture the broad spatial context of a tissue slide, generalize poorly to data from external institutions, and show a marked performance drop when applied to small endoscopic biopsies compared to larger surgical resections. This study addresses these limitations by leveraging the superior contextual modeling and generalization capabilities of transformer architectures.

Methods

The researchers developed a three-stage, fully transformer-based pipeline designed to move beyond the limitations of traditional CNN-based architectures.

1. Feature Extraction with CTransPath

Instead of standard CNN encoders, the pipeline utilizes CTransPath, a Swin Transformer-based encoder. This model was pre-trained using self-supervised learning on a diverse array of pathology images. This approach allows the encoder to learn rich, domain-specific visual features without the need for manual, pixel-level annotations by pathologists.

2. Patch Aggregation via Multi-Headed Self-Attention (MSA)

To process the thousands of small image "tiles" that make up a whole-slide image, the team employed a Transformer network with Multi-Headed Self-Attention. Unlike simple pooling methods (like mean or max pooling) used in previous models, MSA allows the model to relate every tile to every other tile across the entire slide. This enables the model to "contextualize" local features based on the global tissue environment.

3. Weakly Supervised Learning

The system was trained using only patient-level labels (e.g., MSI-high vs. Microsatellite Stable). This weakly supervised approach is highly scalable, as it does not require pathologists to manually outline tumor regions, relying instead on the model to identify the most predictive areas through its attention mechanism.

Experimental Design and Datasets

The study's scale is a key differentiator, utilizing 10 different patient cohorts totaling 9,048 patients. This includes public datasets (TCGA, CPTAC) and several large multicentric cohorts (DACHS, DUSSEL, Epi700, MECC, MUNICH, NLCS, QUASAR, and YCR-BCIP). A critical component of the evaluation was testing the model on a set of 1,502 patients from the YCR-BCIP cohort to assess performance on endoscopic biopsies—a common real-world clinical scenario where tissue samples are limited.

Results

State-of-the-Art Accuracy and Generalization

The transformer-based model demonstrated robust performance across all targets. For MSI prediction on surgical resections, the model achieved a sensitivity of 0.97 and a negative predictive value (NPV) of 0.99. This high NPV is particularly significant for clinical workflows, as it suggests the model could reliably "rule out" MSI-high status, allowing clinicians to focus molecular testing resources on a smaller subset of patients.

Furthermore, the model showed exceptional stability when tested on external cohorts. While traditional CNN-based approaches often see performance drops of up to 0.21 AUROC when moving to new data, the transformer model's performance varied by a maximum of only 0.08 AUROC.

Bridging the Biopsy Gap

One of the study's most significant breakthroughs was the model's performance on endoscopic biopsies. Despite being trained only on surgical resections, the model achieved an AUROC of 0.91 on biopsy samples. This suggests that the transformer architecture learns features that are fundamental to the underlying biology, rather than features specific to the type of tissue preparation.

Data Efficiency and Multi-Target Success

The architecture proved to be remarkably data-efficient, reaching an AUROC > 0.9 with as few as 250 patients. In comparison, previous state-of-the-art methods (such as AttentionMIL) required nearly 4,000 patients to reach similar performance levels. Beyond MSI, the model also successfully predicted other critical biomarkers, including BRAF (AUROC 0.86) and KRAS (AUROC 0.75).

Discussion and Explainability

To ensure the model was making decisions based on relevant biological features, the researchers used "attention rollout" maps. These visualizations confirmed that the model focused on morphologically significant areas, such as lymphocyte aggregates and mucinous regions—both of which are established clinical indicators of MSI. This alignment with known pathology increases the interpretability and trustworthiness of the system.

Despite these successes, the authors identified areas for further refinement. The current model does not explicitly incorporate the spatial coordinates of tiles (positional encoding), which could potentially improve context-awareness. Additionally, while the model is robust, the "generalization gap" caused by stain variability between labs remains a challenge that may require more advanced normalization techniques.

Conclusion

This large-scale study demonstrates that fully transformer-based models can overcome the primary hurdles facing digital biomarkers in colorectal cancer. By providing high-accuracy, clinical-grade predictions on both resections and biopsies, this technology offers a scalable way to accelerate precision oncology and improve patient outcomes through earlier, data-driven treatment decisions.

Summary updated

Abstract

Deep learning (DL) can accelerate the prediction of prognostic biomarkers from routine pathology slides in colorectal cancer (CRC). However, current approaches rely on convolutional neural networks (CNNs) and have mostly been validated on small patient cohorts. Here, we develop a new transformer-based pipeline for end-to-end biomarker prediction from pathology slides by combining a pre-trained transformer encoder with a transformer network for patch aggregation. Our transformer-based approach substantially improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training and evaluating on a large multicenter cohort of over 13,000 patients from 16 colorectal cancer cohorts, we achieve a sensitivity of 0.99 with a negative predictive value of over 0.99 for prediction of microsatellite instability (MSI) on surgical resection specimens. We demonstrate that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem.
BibTeX
@inproceedings{wagner2023transformer,
  author = {Sophia J. Wagner and Daniel Reisenbüchler and Nicholas P. West and Jan Niehues and Jiefu Zhu and Sebastian Foersch and Gregory Patrick Veldhuizen and Philip Quirke and Heike I. Grabsch and Piet A. van den Brandt and Gordon Hutchins and Susan D. Richman and Tanwei Yuan and Rupert Langer and Josien C.A. Jenniskens and Kelly Offermans and Wolfram Mueller and Richard Gray and Stephen B. Gruber and Joel K. Greenson and Gad Rennert and Joseph D. Bonner and Daniel Schmolze and Jitendra Jonnagaddala and Nicholas J. Hawkins and Robyn L. Ward and Dion Morton and Michel Seymour and Laura Magill and Marta Nowak and Jennifer Hay and Viktor H. Koelzer and David N. Church and Christian Matek and Carol Geppert and Chaolong Peng and Cheng Zhi and Xiaoming Ouyang and Jacqueline A. James and Maurice B. Loughrey and Manuel Salto‐Tellez and Hermann Brenner and Michael Hoffmeister and Daniel Truhn and Julia A. Schnabel and Melanie Boxberg and Tingying Peng and Jakob Nikolas Kather and David N. Church and Enric Domingo and Joanne Edwards and Bengt Glimelius and Ismail Gögenür and Andrea Harkin and Jen Hay and Timothy Iveson and Emma Jaeger and Caroline Kelly and Rachel Kerr and Noori Maka and Hannah Morgan and Karin A. Oien and Clare Orange and Claire Palles and Campbell S.D. Roxburgh and Owen J. Sansom and Mark Saunders and Ian Tomlinson},
  title = {{Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study}},
  booktitle = {Cancer},
  year = {2023},
  doi = {10.1016/j.ccell.2023.08.002},
}
Figure