Improving Colorectal Cancer Screening and Risk Assessment through Predictive Modeling on Medical Images and Records

Shuai Jiang, Christina M. Robinson, Joseph Anderson, William Hisey, Lynn Butterly, Arief A. Suriawinata, Saeed HassanpourarXiv 2024 · arXiv · OpenReview

Abstract

Colonoscopy screening effectively identifies and removes polyps before they progress to colorectal cancer (CRC), but current follow-up guidelines rely primarily on histopathological features, overlooking other important CRC risk factors. Variability in polyp characterization among pathologists also hinders consistent surveillance decisions. Advances in digital pathology and deep learning enable the integration of pathology slides and medical records for more accurate CRC risk prediction. Using data from the New Hampshire Colonoscopy Registry, including longitudinal follow-up, we adapted a transformer-based model for histopathology image analysis to predict 5-year CRC risk. We further explored multi-modal fusion strategies to combine clinical records with deep learning-derived image features. Training the model to predict intermediate clinical variables improved 5-year CRC risk prediction (AUC = 0.630) compared to direct prediction (AUC = 0.615, p = 0.013). Incorporating both imaging and non-imaging data, without requiring manual slide review, further improved performance (AUC = 0.674) compared to traditional features from colonoscopy and microscopy reports (AUC = 0.655, p = 0.001). These results highlight the value of integrating diverse data modalities with computational methods to enhance CRC risk stratification.

Introduction

Colorectal cancer (CRC) remains a significant cause of cancer-related mortality worldwide, despite being highly preventable through timely screening and polyp removal. Current clinical guidelines for post-colonoscopy surveillance are largely based on the manual assessment of polyp characteristics—such as size, type, and location—by pathologists. However, this approach is limited by significant inter-observer variability and the exclusion of broader patient data, including lifestyle factors and family history, which are known to influence cancer progression. To address these limitations, this research presents a multi-modal deep learning framework that integrates whole-slide pathology images (WSIs) with longitudinal electronic medical records (EMRs) to provide a more precise, personalized 5-year CRC risk assessment.

Methodology

The proposed pipeline utilizes a multi-stage deep learning approach to synthesize diverse data sources into a unified risk score.

Image Analysis with MaskHIT

At the core of the vision component is MaskHIT, a transformer-based model adapted for gigapixel WSIs. Unlike traditional convolutional neural networks, MaskHIT employs a Masked AutoEncoder technique to capture complex spatial relationships between tissue patches, allowing the model to learn robust representations of histological architecture.

Guided Attention and Intermediate Variables

A key innovation in this study is the use of "guided attention." Rather than training the model to predict long-term CRC risk directly from raw images, the researchers first trained it to identify intermediate clinical variables, such as polyp size and the presence of advanced adenomas. This intermediate step acts as a scaffold, focusing the model's attention on histologically relevant regions before it attempts the more complex task of 5-year risk prediction.

Multi-modal Fusion Strategies

The framework explores two primary methods for integrating image-derived features with clinical data (e.g., BMI, smoking history, and endoscopy reports):

  1. Feature-level (Early) Fusion: Concatenating data vectors from both modalities into a single input for the final classifier.
  2. Decision-level (Late) Fusion: Independently calculating risk probabilities from the image and clinical models and then averaging the results.
Overview of the multi-modal deep learning pipeline for CRC risk assessment.

Results

The study utilized data from the New Hampshire Colonoscopy Registry (NHCR), encompassing 2,393 patients with 5-year longitudinal follow-up. The integrated multi-modal model achieved a significantly higher predictive accuracy (AUC = 0.674) compared to models relying solely on traditional features extracted from colonoscopy and microscopy reports (AUC = 0.655, p = 0.001).

Furthermore, the "guided" image model (AUC = 0.630) outperformed the direct image-to-risk model (AUC = 0.615), validating the importance of incorporating domain-specific intermediate variables. Feature importance analysis using Shapley values revealed that the AI-generated WSI risk score was the third most significant predictor of future cancer, following only the number of adenomas and patient age. This suggests that deep learning can uncover prognostic information within tissue architecture that is currently overlooked during manual pathological review.

Discussion and Limitations

While the results demonstrate the potential for AI to refine CRC surveillance, several challenges remain. The current dataset is derived from a single statewide registry with limited demographic diversity, necessitating further validation across more varied populations. Additionally, while interpretability tools like attention maps and Shapley values provide some insight, the "black box" nature of deep learning remains a hurdle for clinical implementation. Future work will focus on prospective clinical trials to evaluate the real-world impact of this model on patient outcomes and healthcare costs.

Conclusion

By fusing digital pathology with longitudinal clinical records through a guided transformer-based architecture, this research provides a path toward more accurate and personalized colorectal cancer screening. Moving beyond the traditional pathologist's lens, these computational methods offer a scalable way to integrate complex, multi-modal data for improved patient risk stratification.

Summary updated

Abstract

Colonoscopy screening effectively identifies and removes polyps before they progress to colorectal cancer (CRC), but current follow-up guidelines rely primarily on histopathological features, overlooking other important CRC risk factors. Variability in polyp characterization among pathologists also hinders consistent surveillance decisions. Advances in digital pathology and deep learning enable the integration of pathology slides and medical records for more accurate CRC risk prediction. Using data from the New Hampshire Colonoscopy Registry, including longitudinal follow-up, we adapted a transformer-based model for histopathology image analysis to predict 5-year CRC risk. We further explored multi-modal fusion strategies to combine clinical records with deep learning-derived image features. Training the model to predict intermediate clinical variables improved 5-year CRC risk prediction (AUC = 0.630) compared to direct prediction (AUC = 0.615, p = 0.013). Incorporating both imaging and non-imaging data, without requiring manual slide review, further improved performance (AUC = 0.674) compared to traditional features from colonoscopy and microscopy reports (AUC = 0.655, p = 0.001). These results highlight the value of integrating diverse data modalities with computational methods to enhance CRC risk stratification.
BibTeX
@misc{jiang2024improving,
  author = {Shuai Jiang and Christina M. Robinson and Joseph Anderson and William Hisey and Lynn Butterly and Arief A. Suriawinata and Saeed Hassanpour},
  title = {{Improving Colorectal Cancer Screening and Risk Assessment through Predictive Modeling on Medical Images and Records}},
  year = {2024},
  eprint = {2410.09880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.09880},
}
Figure