Fully transformer-based biomarker prediction from colorectal cancer histology: a large-scale multicentric study

Sophia J. Wagner, Daniel Reisenbüchler, Nicholas P. West, Jan Moritz Niehues, Gregory Patrick Veldhuizen, Philip Quirke, Heike Irmgard Grabsch, Piet A. van den Brandt, Gordon G. A. Hutchins, Susan D. Richman, Tanwei Yuan, Rupert Langer, Josien Christina Anna Jenniskens, Kelly Offermans, Wolfram Müller, Richard Gray, Stephen B. Gruber, Joel K. Greenson, Gad Rennert, Joseph D. Bonner, Daniel Schmolze, Jacqueline A. James, Maurice B. Loughrey, Manuel Salto-Tellez, Hermann Brenner, Michael Hoffmeister, Daniel Truhn, Julia A. Schnabel, Melanie Boxberg, Tingying Peng, Jakob Nikolas KatherarXiv 2023 · DOI · OpenReview

Abstract

Precision oncology for colorectal cancer (CRC) is currently limited by the time and cost associated with molecular assays for biomarkers like Microsatellite Instability (MSI). While deep learning has shown potential in predicting these markers from H&E pathology slides, traditional convolutional neural networks (CNNs) often struggle with global context and generalization across different clinical settings. This study introduces a fully transformer-based pipeline trained on a massive multicentric dataset of over 9,000 patients. The model achieves state-of-the-art performance, demonstrating high sensitivity and, crucially, clinical-grade accuracy on endoscopic biopsies even when trained solely on surgical resections. These findings suggest a path toward immediate, automated biomarker screening at the point of initial diagnosis.

Introduction

The identification of genetic biomarkers such as Microsatellite Instability (MSI) and mutations in BRAF or KRAS is essential for tailoring treatment in colorectal cancer. However, the current gold standard—molecular testing via PCR or next-generation sequencing—can take weeks to process. This delay can postpone the initiation of targeted therapies or immunotherapy.

Digital pathology offers a solution by using deep learning (DL) to predict molecular status directly from routine hematoxylin and eosin (H&E) stained slides. Despite this potential, existing DL methods face significant hurdles: they often fail to capture the broad spatial context of a tissue slide, generalize poorly to data from external institutions, and show a marked performance drop when applied to small endoscopic biopsies compared to larger surgical resections. This study addresses these limitations by leveraging the superior contextual modeling and generalization capabilities of transformer architectures.

Methods

The researchers developed a three-stage, fully transformer-based pipeline designed to move beyond the limitations of traditional CNN-based architectures.

1. Feature Extraction with CTransPath

Instead of standard CNN encoders, the pipeline utilizes CTransPath, a Swin Transformer-based encoder. This model was pre-trained using self-supervised learning on a diverse array of pathology images. This approach allows the encoder to learn rich, domain-specific visual features without the need for manual, pixel-level annotations by pathologists.

2. Patch Aggregation via Multi-Headed Self-Attention (MSA)

To process the thousands of small image "tiles" that make up a whole-slide image, the team employed a Transformer network with Multi-Headed Self-Attention. Unlike simple pooling methods (like mean or max pooling) used in previous models, MSA allows the model to relate every tile to every other tile across the entire slide. This enables the model to "contextualize" local features based on the global tissue environment.

3. Weakly Supervised Learning

The system was trained using only patient-level labels (e.g., MSI-high vs. Microsatellite Stable). This weakly supervised approach is highly scalable, as it does not require pathologists to manually outline tumor regions, relying instead on the model to identify the most predictive areas through its attention mechanism.

Experimental Design and Datasets

The study's scale is a key differentiator, utilizing 10 different patient cohorts totaling 9,048 patients. This includes public datasets (TCGA, CPTAC) and several large multicentric cohorts (DACHS, DUSSEL, Epi700, MECC, MUNICH, NLCS, QUASAR, and YCR-BCIP). A critical component of the evaluation was testing the model on a set of 1,502 patients from the YCR-BCIP cohort to assess performance on endoscopic biopsies—a common real-world clinical scenario where tissue samples are limited.

Results

State-of-the-Art Accuracy and Generalization

The transformer-based model demonstrated robust performance across all targets. For MSI prediction on surgical resections, the model achieved a sensitivity of 0.97 and a negative predictive value (NPV) of 0.99. This high NPV is particularly significant for clinical workflows, as it suggests the model could reliably "rule out" MSI-high status, allowing clinicians to focus molecular testing resources on a smaller subset of patients.

Furthermore, the model showed exceptional stability when tested on external cohorts. While traditional CNN-based approaches often see performance drops of up to 0.21 AUROC when moving to new data, the transformer model's performance varied by a maximum of only 0.08 AUROC.

Bridging the Biopsy Gap

One of the study's most significant breakthroughs was the model's performance on endoscopic biopsies. Despite being trained only on surgical resections, the model achieved an AUROC of 0.91 on biopsy samples. This suggests that the transformer architecture learns features that are fundamental to the underlying biology, rather than features specific to the type of tissue preparation.

Data Efficiency and Multi-Target Success

The architecture proved to be remarkably data-efficient, reaching an AUROC > 0.9 with as few as 250 patients. In comparison, previous state-of-the-art methods (such as AttentionMIL) required nearly 4,000 patients to reach similar performance levels. Beyond MSI, the model also successfully predicted other critical biomarkers, including BRAF (AUROC 0.86) and KRAS (AUROC 0.75).

Discussion and Explainability

To ensure the model was making decisions based on relevant biological features, the researchers used "attention rollout" maps. These visualizations confirmed that the model focused on morphologically significant areas, such as lymphocyte aggregates and mucinous regions—both of which are established clinical indicators of MSI. This alignment with known pathology increases the interpretability and trustworthiness of the system.

Despite these successes, the authors identified areas for further refinement. The current model does not explicitly incorporate the spatial coordinates of tiles (positional encoding), which could potentially improve context-awareness. Additionally, while the model is robust, the "generalization gap" caused by stain variability between labs remains a challenge that may require more advanced normalization techniques.

Conclusion

This large-scale study demonstrates that fully transformer-based models can overcome the primary hurdles facing digital biomarkers in colorectal cancer. By providing high-accuracy, clinical-grade predictions on both resections and biopsies, this technology offers a scalable way to accelerate precision oncology and improve patient outcomes through earlier, data-driven treatment decisions.

Summary updated

Abstract

Background: Deep learning (DL) can extract predictive and prognostic biomarkers from routine pathology slides in colorectal cancer. For example, a DL test for the diagnosis of microsatellite instability (MSI) in CRC has been approved in 2022. Current approaches rely on convolutional neural networks (CNNs). Transformer networks are outperforming CNNs and are replacing them in many applications, but have not been used for biomarker prediction in cancer at a large scale. In addition, most DL approaches have been trained on small patient cohorts, which limits their clinical utility. Methods: In this study, we developed a new fully transformer-based pipeline for end-to-end biomarker prediction from pathology slides. We combine a pre-trained transformer encoder and a transformer network for patch aggregation, capable of yielding single and multi-target prediction at patient level. We train our pipeline on over 9,000 patients from 10 colorectal cancer cohorts. Results: A fully transformer-based approach massively improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training on a large multicenter cohort, we achieve a sensitivity of 0.97 with a negative predictive value of 0.99 for MSI prediction on surgical resection specimens. We demonstrate for the first time that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem. Interpretation: A fully transformer-based end-to-end pipeline trained on thousands of pathology slides yields clinical-grade performance for biomarker prediction on surgical resections and biopsies. Our new methods are freely available under an open source license.
BibTeX
@misc{wagner2023fully,
  author = {Sophia J. Wagner and Daniel Reisenbüchler and Nicholas P. West and Jan Moritz Niehues and Gregory Patrick Veldhuizen and Philip Quirke and Heike Irmgard Grabsch and Piet A. van den Brandt and Gordon G. A. Hutchins and Susan D. Richman and Tanwei Yuan and Rupert Langer and Josien Christina Anna Jenniskens and Kelly Offermans and Wolfram Müller and Richard Gray and Stephen B. Gruber and Joel K. Greenson and Gad Rennert and Joseph D. Bonner and Daniel Schmolze and Jacqueline A. James and Maurice B. Loughrey and Manuel Salto-Tellez and Hermann Brenner and Michael Hoffmeister and Daniel Truhn and Julia A. Schnabel and Melanie Boxberg and Tingying Peng and Jakob Nikolas Kather},
  title = {{Fully transformer-based biomarker prediction from colorectal cancer histology: a large-scale multicentric study}},
  year = {2023},
  eprint = {2301.09617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.09617},
}
Figure