RNA-to-image multi-cancer synthesis using cascaded diffusion models

Francisco Carrillo‐Pérez, Marija Pizurica, Yuanning Zheng, Tarak Nandi, Ravi Madduri, Jeanne Shen, Olivier GevaertbioRxiv 2023 · DOI

TL;DR

This paper introduces RNA-CDM, a novel cascaded diffusion model that synthesizes realistic whole-slide images (WSIs) of multi-cancer tissues directly from RNA sequencing (RNA-Seq) data. This approach addresses the critical challenge of data scarcity in biomedical research by generating high-quality synthetic data, which can then be used to train more robust machine learning models for cancer diagnosis and prognosis.

What Problem Does This Paper Address?

Acquiring large, diverse, and multimodal datasets for biomedical research, especially in cancer, is incredibly challenging and expensive. This data scarcity hinders the development of powerful machine learning models that could aid in diagnosis and treatment. While previous work has focused on generating synthetic data for single modalities (like just images or just RNA-Seq) or for single cancer types, there's a significant need for methods that can bridge the gap between different data types (RNA-Seq and WSIs) and handle multiple cancer types simultaneously. This paper aims to solve this by creating realistic WSI "tiles" from a patient's RNA-Seq profile, effectively generating a visual representation of the tumor's molecular state.

Methods

The core of the RNA-CDM approach involves two main stages:

  1. Latent Embedding Generation (β-VAE): To handle the high dimensionality of RNA-Seq data, the authors first use a β-Variational Autoencoder (β-VAE). This model learns to compress the gene expression profiles of patients into a lower-dimensional "latent space." This latent representation captures the essential molecular characteristics of different cancer types, making it suitable for conditioning image generation.
  2. RNA-to-Image Synthesis (Cascaded Diffusion Models): Inspired by recent advances in text-to-image generation, the researchers employ a cascaded diffusion model. This model takes the latent representation from the β-VAE as input and generates realistic H&E-stained WSI tiles. The "cascaded" aspect means it uses a two-stage diffusion process: a lower-resolution diffusion model generates a base image, which is then upscaled by a super-resolution diffusion model to produce the final high-resolution tile. This architecture allows for generating diverse and detailed images.

Datasets

  • Training Data: The primary dataset used is from The Cancer Genome Atlas (TCGA) project, which provides paired RNA-Seq and WSI data for multiple cancer types.
  • For β-VAE training: 12 cancer types from TCGA.
  • For RNA-CDM training and validation: 5 cancer types (LUAD, GBM, KIRP, CESC, COAD) were selected based on morphological differences and sample availability.
  • Generalization Experiments: Two external datasets from GEO (GSE50760 for colorectal cancer and GSE226069 for lung cancer) were used to test the model's ability to generalize to unseen data.
  • Microsatellite Instability Prediction: Data from Kather et al. was used for a specific downstream task.

Key Insights and Ideas

  • Bridging Modalities: The paper successfully demonstrates that it's possible to generate realistic histological images directly from molecular data (RNA-Seq), effectively bridging the gap between these two crucial modalities in cancer research.
  • Multi-Cancer Synthesis: Unlike previous methods that often focused on single cancer types, RNA-CDM can generate synthetic images for multiple cancer types using a single architecture, making it more practical and scalable.
  • Latent Space Power: The use of a β-VAE to create a compressed latent representation of RNA-Seq data is key. This allows the diffusion model to learn complex relationships between gene expression and tissue morphology without being overwhelmed by the raw data's dimensionality.
  • Synthetic Data for Pretraining: The generated synthetic WSI tiles can be effectively used to pretrain machine learning models, significantly improving their performance on real-world classification tasks, even when the synthetic dataset is smaller than the real one.

Key Results

  • Realistic Image Generation: RNA-CDM generates visually realistic H&E-stained WSI tiles for five different cancer types, capturing distinct morphological characteristics.
  • Cell Distribution Preservation: State-of-the-art cell segmentation models (HoverNet) successfully identify important cell types in the synthetic tiles, and the distribution of these cell types closely matches that of real tiles.
  • Correlation with Bulk RNA-Seq: The cell fractions observed in the synthetic tiles significantly correlate with the cell fractions predicted from bulk RNA-Seq data, indicating that the synthetic data preserves key biological information.
  • Improved Model Performance: Using synthetic data for pretraining machine learning models (e.g., ResNet18 for classification) led to improved accuracy and F1-scores compared to training from scratch, without introducing significant new biases.
  • Generalization: The model demonstrated the ability to generate synthetic tiles for cancer types not included in its training set, showcasing its generalization capabilities.
  • Quantitative Metrics: The generated tiles achieved a FID50k score of 23.36, an IS50k of 3.19, and a KID50k of 0.015, indicating good quality and diversity.

Limitations and Future Work

  • Sampling Speed: While diffusion models excel at quality, their sampling speed can be a bottleneck. The authors mention that future research could address this, potentially by exploring faster sampling methods.
  • TCGA Data Reliance: The model was trained on TCGA data, which has specific inclusion criteria (e.g., at least 60% tumor nuclei). Future work could explore training on more diverse datasets.
  • Spatial Transcriptomics: The authors express excitement about emerging spatial transcriptomics technologies, suggesting that integrating this type of data could further enhance the capabilities of models like RNA-CDM.
  • Downstream Task Performance: While synthetic data improved pretraining, the paper also notes that substituting real data with synthetic data did not negatively impact classification performance, suggesting its potential for data augmentation in low-data scenarios.
  • Potential for Novel Discoveries: The ability to link gene expression to morphology could potentially uncover novel morphologic features associated with clinically relevant molecular states that are not easily recognizable by human eyes.

Summary updated

Abstract

Data scarcity presents a significant obstacle in the field of biomedicine, where acquiring diverse and sufficient datasets can be costly and challenging. Synthetic data generation offers a potential solution to this problem by expanding dataset sizes, thereby enabling the training of more robust and generalizable machine learning models. Although previous studies have explored synthetic data generation for cancer diagnosis, they have predominantly focused on single modality settings, such as whole-slide image tiles or RNA-Seq data. To bridge this gap, we propose a novel approach, RNA-Cascaded-Diffusion-Model or RNA-CDM, for performing RNA-to-image synthesis in a multi-cancer context, drawing inspiration from successful text-to-image synthesis models used in natural images. In our approach, we employ a variational auto-encoder to reduce the dimensionality of a patient's gene expression profile, effectively distinguishing between different types of cancer. Subsequently, we employ a cascaded diffusion model to synthesize realistic whole-slide image tiles using the latent representation derived from the patient's RNA-Seq data. Our results demonstrate that the generated tiles accurately preserve the distribution of cell types observed in real-world data, with state-of-the-art cell identification models successfully detecting important cell types in the synthetic samples. Furthermore, we illustrate that the synthetic tiles maintain the cell fraction observed in bulk RNA-Seq data and that modifications in gene expression affect the composition of cell types in the synthetic tiles. Next, we utilize the synthetic data generated by RNA-CDM to pretrain machine learning models and observe improved performance compared to training from scratch. Our study emphasizes the potential usefulness of synthetic data in developing machine learning models in sarce-data settings, while also highlighting the possibility of imputing missing data modalities by leveraging the available information. In conclusion, our proposed RNA-CDM approach for synthetic data generation in biomedicine, particularly in the context of cancer diagnosis, offers a novel and promising solution to address data scarcity. By generating synthetic data that aligns with real-world distributions and leveraging it to pretrain machine learning models, we contribute to the development of robust clinical decision support systems and potential advancements in precision medicine.
BibTeX
@misc{carrilloperez2023rna,
  author = {Francisco Carrillo‐Pérez and Marija Pizurica and Yuanning Zheng and Tarak Nandi and Ravi Madduri and Jeanne Shen and Olivier Gevaert},
  title = {{RNA-to-image multi-cancer synthesis using cascaded diffusion models}},
  year = {2023},
  doi = {10.1101/2023.01.13.523899},
  howpublished = {bioRxiv},
}
Figure