Abstract
Survival analysis using pathology images poses a considerable challenge, as it requires the localization of relevant information from the multitude of tiles within whole slide images (WSIs). Current methods typically resort to a two-stage approach, where a pre-trained network extracts features from tiles, which are then used by survival models. This process, however, does not optimize the survival models in an end-to-end manner, and the pre-extracted features may not be ideally suited for survival prediction. To address this limitation, we present a novel end-to-end Visual Prompt Tuning framework for survival analysis, named VPTSurv. VPTSurv refines feature embeddings through an efficient encoder-decoder framework. The encoder remains fixed while the framework introduces tunable visual prompts and adaptors, thus permitting end-to-end training specifically for survival prediction by optimizing only the lightweight adaptors and the decoder. Moreover, the versatile VPTSurv framework accommodates multi-source information as prompts, thereby enriching the survival model. VPTSurv achieves substantial increases of 8.7% and 12.5% in the C-index on two immunohistochemical pathology image datasets. These significant improvements highlight the transformative potential of the end-to-end VPT framework over traditional two-stage methods.
Introduction
Predicting patient survival from Whole Slide Images (WSIs) is often described as a "needle in a haystack" problem. A single WSI can contain billions of pixels, making it impossible to process the entire image in GPU memory for standard end-to-end training. Consequently, the industry standard has shifted toward a two-stage pipeline: offline feature extraction followed by a separate aggregation model, such as Multiple Instance Learning.
However, this approach has a fundamental flaw: the features extracted in the first stage are typically optimized for general image recognition or general pathology tasks, not specifically for survival analysis. Since the feature extractor is frozen during the second stage, the model cannot refine its visual representations to identify prognostic patterns. True end-to-end optimization is the ideal, but memory constraints have historically made it prohibitive.
Methods: The VPTSurv Framework
To bridge this gap, we introduce VPTSurv, a framework that leverages Visual Prompt Tuning (VPT) to enable efficient end-to-end training. Instead of updating the millions of parameters in a heavy Vision Transformer (ViT) encoder, VPTSurv keeps the encoder frozen and introduces lightweight, learnable "adaptors" into each layer.
As shown in the architecture above, only these adaptors and the final survival decoder are trained. This strategy significantly reduces the memory footprint while allowing the model to "steer" the pre-trained backbone toward survival-relevant features. Furthermore, VPTSurv incorporates multi-source information as prompts to enrich the model's context:
- Scale Information: Contextual features from lower-resolution views of the slide.
- Structural Information: Global tissue patterns captured via Fast Fourier Transform (FFT).
Experimental Results
The framework was evaluated on two immunohistochemical (IHC) pancreatic cancer datasets: IHC-CD4 (564 patients) and IHC-CD8 (564 patients). These stains highlight specific immune cell markers critical to the tumor microenvironment.
VPTSurv was compared against state-of-the-art two-stage methods, including TransMIL and DeepAttnMISL. The primary metric, the Concordance Index (C-index), measures the model's ability to correctly rank patient survival times.
As detailed in the results table, VPTSurv achieved a C-index of 59.93% on IHC-CD4 and 61.10% on IHC-CD8, representing an 8.7% and 12.5% relative improvement over the best two-stage baselines, respectively. Ablation studies further confirmed that integrating multi-source prompts (structure and scale) consistently outperformed using visual prompts alone.
Conclusion
VPTSurv demonstrates that end-to-end training for WSI survival analysis is not only feasible but superior to traditional two-stage methods. By tuning approximately 1% of the model's parameters, we can effectively adapt large-scale pre-trained models to specialized medical tasks. Future work will focus on expanding the framework to include genomic and clinical data as prompts, as well as validating the approach on more diverse cancer types and H&E-stained slides.
Summary updated