Attention-based deep learning for analysis of pathology images and gene expression data in lung squamous premalignant lesions

Lingyi Xu, Yohana Kefella, Yichi Zhang, Regan D. Conrad, Kelley Anderson, Kostyantyn Krysan, Gang Liu, Erin E. Kane, Adam Pennycuick, Sam M. Janes, Mary E. Reid, Eric Burks, Ehab Billatos, Sarah A. Mazzilli, Vijaya B. Kolachalama, Jennifer BeanemedRxiv 2025 · DOI

TL;DR

Researchers have developed a multimodal transformer-based deep learning framework that combines pathology images and gene expression data to identify lung cancer precursors. By using a "flexible fusion" approach, the model significantly outperforms single-modality methods and remains effective even when one type of data is missing, offering a more robust way to track the progression of lung squamous premalignant lesions.

What Problem Does This Paper Address?

Lung squamous cell carcinoma doesn't appear overnight; it develops through a series of "premalignant lesions" (PMLs) that progress from normal tissue to hyperplasia, dysplasia, and eventually invasive cancer. However, grading these lesions is notoriously subjective. Pathologists often disagree on where a lesion falls on this spectrum (high inter-observer variability), which makes it difficult to consistently identify patients at the highest risk.

While molecular data (gene expression) and structural data (H&E images) both provide clues, they are rarely used together in a unified computational framework. Furthermore, in real-world clinical settings, you might have a biopsy slide but no sequencing data, or vice versa. This paper tackles the challenge of creating a reliable, multimodal tool that can handle the messy reality of incomplete clinical data.

Methods

The authors built a transformer-based multimodal framework designed to ingest two very different types of data:

  1. Image Branch: Digitized H&E whole slide images (WSIs) are broken into patches. These patches are converted into feature vectors using CTransPath (a self-supervised foundation model) and then organized into a graph. A Graph Convolutional Network (GCN) with minCUT pooling processes these to capture spatial relationships between cells.
  2. Gene Branch: Bulk gene expression (GE) data is passed through a fully connected network.

The "secret sauce" is the Flexible Fusion strategy. During training, the authors used "modality masks" to randomly hide either the image or the gene data. This forced the transformer encoder to learn a shared, complementary representation of the disease. Because of this training style, the model can perform inference on a sample even if only one data modality is available, without a total loss of predictive power.

Datasets

The study leveraged an impressive array of seven different cohorts to ensure the model wasn't just "memorizing" one specific lab's data:

  • Training/Internal Testing: Lung Pre-Cancer Atlas (PCA) Datasets 1 & 2, the Colorado SPORE cohort, and the Pennycuick (UCLH) study.
  • External Validation: The DECAMP-1 (military personnel) cohort and the Jules Bordet Institute cohort.
  • Surrogate Data: Bronchial brushings (cells collected from "normal-appearing" areas) were used to test if the model could detect a "field effect" of cancerization.

Key Insights and Ideas

  • Multimodal Synergy: The most striking finding was that training on both images and genes improved performance even when the model was eventually tested on only one modality. It seems the model uses the gene data during training to "teach" the image branch which visual features actually matter, and vice versa.
  • The Probability Spectrum: Although the model was trained on a binary task (Dysplasia vs. Non-Dysplasia), its output probabilities actually mapped beautifully to the full histologic spectrum. It effectively turned a subjective category into a continuous "risk score."
  • Biological "Aha" Moments: The model didn't just find random patterns; it identified gene sets associated with the cell cycle and proliferation (like PLK1 and AURKA) and the downregulation of Notch signaling. These are known drivers of lung cancer, providing a "sanity check" that the AI is looking at relevant biology.
  • Field Cancerization: The model could predict the presence of dysplasia elsewhere in the lung by looking at gene expression from "normal-looking" bronchial brushings, suggesting it could be used as a wide-area screening tool.

Key Results

  • Performance Boost: On external WSI data, the multimodal-trained model achieved an AUROC of 0.761, compared to just 0.690 for a model trained only on images.
  • Gene Superiority: On external gene expression data, the multimodal approach reached an AUROC of 0.890, outperforming the gene-only model (0.816).
  • Flexible Fusion Success: The final "flexible" model achieved an AUROC of 0.809 on external images and 0.903 on external gene data, proving that you don't need perfectly paired datasets to build a high-performing clinical tool.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Pathology Gold Standard: The model was trained on labels from individual studies rather than a single, centralized "consensus" review by a panel of pathologists. This means some "noise" from human subjectivity is still baked into the ground truth.
  • Biopsy Size: The WSIs came from forceps biopsies, which are tiny. This might limit the model's ability to see the "big picture" of the lung architecture.
  • Static vs. Dynamic: This study looked at a single point in time. Future work needs to use longitudinal data to see if these AI scores can actually predict which specific lesions will turn into cancer years down the line.
  • Next-Gen Data: The authors suggest that incorporating single-cell sequencing or spatial transcriptomics could provide even deeper insights into the "immune microenvironment" of these lesions.

Summary updated

Abstract

Molecular and cellular alterations to the normal pseudostratified columnar bronchial epithelium results in the development of bronchial premalignant lesions representing a spectrum of histology from normal to hyperplasia, metaplasia, dysplasia (mild, moderate, and severe), carcinoma in situ and invasive carcinoma. Several studies have identified molecular alterations associated with lesion histology and progression. The broad and continuous spectrum of histologic and molecular changes makes reproducible stratification of lesions across multiple studies challenging. Here we propose a transformer-based framework that flexibly utilizes transcriptomic and histologic patterns to distinguish lesions with bronchial dysplasia or worse from normal, hyperplasia, and metaplasia. We leveraged H&E whole slide images (WSIs) of endobronchial biopsies and bulk gene expression data (GE) from previously published studies and on-going lung precancer atlas efforts obtained from patients as high-risk for lung cancer. Models trained using both WSIs and GE compared to a single data modality had higher performance. On an external testing dataset of WSIs, the area under the ROC curve (AUROC) of the model trained on WSIs plus GE was 0.761±0.015 compared to 0.690±0.027 for model trained on WSIs. On external testing datasets of GE, the AUROC of the model trained on WSIs plus GE was 0.890±0.023 versus 0.816±0.032 for a model trained on GE. Based on these results, we leveraged data across 4 studies to train a flexible fusion model that allows one or both data modalities to be used in training. The model achieved an AUROC of 0.809±0.036 on external testing WSIs data and 0.903±0.022 on external testing GE data. Despite model training on a binary label, model probabilities are associated with histologic grade and the model identifies gene expression alterations associated with bronchial dysplasia across multiple studies. This framework maps bronchial premalignant lesions that contain at least one data modality into a spectrum of disease. In the future, a framework trained on multiple data modalities may be useful in predicting premalignant disease severity, progression, and interception agent efficacy.
BibTeX
@misc{xu2025attention,
  author = {Lingyi Xu and Yohana Kefella and Yichi Zhang and Regan D. Conrad and Kelley Anderson and Kostyantyn Krysan and Gang Liu and Erin E. Kane and Adam Pennycuick and Sam M. Janes and Mary E. Reid and Eric Burks and Ehab Billatos and Sarah A. Mazzilli and Vijaya B. Kolachalama and Jennifer Beane},
  title = {{Attention-based deep learning for analysis of pathology images and gene expression data in lung squamous premalignant lesions}},
  year = {2025},
  doi = {10.1101/2025.06.06.25328492},
  howpublished = {medRxiv},
}
Figure