"A net for everyone": fully personalized and unsupervised neural networks trained with longitudinal data from a single patient

Christian Strack, Kelsey L. Pomykala, Heinz-Peter Schlemmer, Jan Egger, Jens Kleesiek

Introduction

Deep learning in medical imaging typically faces a significant "data bottleneck." State-of-the-art models generally require thousands of expertly labeled images to perform tasks like tumor segmentation. In clinical practice, gathering such datasets is hindered by strict privacy regulations (e.g., GDPR or HIPAA) and the immense time required for radiologists to manually annotate scans.

Furthermore, monitoring tumor growth over time usually necessitates "co-registration"—the mathematical alignment of scans from different time points. This process is notoriously difficult when tumors cause "mass effects" that deform brain tissue or when post-surgical changes alter a patient's anatomy. To address these challenges, researchers have developed a "personalized" deep learning approach that trains a neural network using longitudinal MRI data from just one patient, creating a privacy-safe and registration-free method.

Methodology: A Personalized WGAN

The core of this approach is an unsupervised Wasserstein Generative Adversarial Network (WGAN) architecture. Unlike traditional models that learn from a population, this network is trained from scratch for every individual patient using only their specific longitudinal scans.

The unsupervised WGAN framework for longitudinal tumor monitoring.

The WGAN consists of two adversarial sub-models:

  1. The Generator (U-Net): This model takes a patient’s initial scan ($t1$) and learns to generate a "difference map." When this map is added back to $t1$, it produces a synthetic version of the subsequent scan ($t2$).
  2. The Critic: This model attempts to distinguish between the actual $t2$ scan and the synthetic version produced by the generator.
The U-Net architecture employed as the generator sub-model.

Through this adversarial process, the generator is forced to learn the precise structural changes—such as tumor progression or regression—that occurred between the two time points.

Key Innovations

  • Zero-Shot/Personalized Training: By training on a single patient's data, the model inherently protects privacy as data never needs to be pooled into a central repository.
  • Registration-Free Learning: The authors utilized heavy data augmentation (randomly rotating and shifting images during training). This forces the network to learn the underlying structural differences and spatial representations regardless of orientation, eliminating the need for traditional pre-alignment.
  • Unsupervised Nature: No manual "ground truth" labels or expert annotations were used during the training process, relying instead on the inherent information within the longitudinal scans.

Results and Clinical Utility

The method was evaluated using two longitudinal datasets of patients with glioblastoma multiforme (GBM): a local dataset of 15 patients and a public dataset (TCIA) of 17 patients. For evaluation purposes, a state-of-the-art supervised segmentation network (nnU-Net) served as the benchmark.

  • Diagnostic Accuracy: The model achieved an AUC-score of 0.87 for detecting tumor change, performing comparably to supervised models trained on massive external datasets.
  • Clinical Correlation: When compared against the standard RANO (Response Assessment in Neuro-Oncology) criteria, the personalized nets achieved 65.5% total accuracy in classifying patients into categories of "Response," "Stable Disease," or "Progression."
  • Radiological Support: The generated heatmaps successfully highlighted key regions of change, acting as a "second pair of eyes" to help radiologists quickly identify areas of concern.

Limitations and Future Outlook

While promising, the model does face challenges. It is designed to detect any change, meaning structural shifts (like a "midline shift" where a tumor pushes brain tissue) can sometimes be incorrectly flagged as growth. Additionally, the model shows sensitivity to noise at the brain's edges and near fluid-filled ventricles.

Because this "N=1" approach does not benefit from the broad knowledge of population-scale datasets, it is best viewed as a specialized tool for longitudinal monitoring. Future work may expand this methodology to other medical tasks, such as tracking aortic aneurysms or other progressive anatomical changes.

Summary updated

Abstract

Background With the rise in importance of personalized medicine and deep learning, we combine the two to create personalized neural networks. The aim of the study is to show a proof of concept that data from just one patient can be used to train deep neural networks to detect tumor progression in longitudinal datasets. Methods Two datasets with 64 scans from 32 patients with glioblastoma multiforme (GBM) were evaluated in this study. The contrast-enhanced T1w sequences of brain magnetic resonance imaging (MRI) images were used. We trained a neural network for each patient using just two scans from different timepoints to map the difference between the images. The change in tumor volume can be calculated with this map. The neural networks were a form of a Wasserstein-GAN (generative adversarial network), an unsupervised learning architecture. The combination of data augmentation and the network architecture allowed us to skip the co-registration of the images. Furthermore, no additional training data, pre-training of the networks or any (manual) annotations are necessary. Results The model achieved an AUC-score of 0.87 for tumor change. We also introduced a modified RANO criteria, for which an accuracy of 66% can be achieved. Conclusions We show a novel approach to deep learning in using data from just one patient to train deep neural networks to monitor tumor change. Using two different datasets to evaluate the results shows the potential to generalize the method.
BibTeX
@article{strack2023net,
  author = {Christian Strack and Kelsey L. Pomykala and Heinz-Peter Schlemmer and Jan Egger and Jens Kleesiek},
  title = {{"A net for everyone": fully personalized and unsupervised neural networks trained with longitudinal data from a single patient}},
  journal = {BMC Medical Imaging},
  year = {2023},
  doi = {10.1186/s12880-023-01128-w},
}
Figure