Patient-specific fine-tuning of convolutional neural networks for follow-up lesion quantification

Mariëlle J. A. Jansen, Hugo J. Kuijf, Ashis Kumar Dhara, Nick A. Weaver, Geert Jan Biessels, Robin Strand, Josien P. W. PluimJournal of Medical Imaging 2020 · DOI

TL;DR

This paper introduces "patient-specific fine-tuning" (FT) for convolutional neural networks (CNNs) to improve lesion quantification in medical imaging. By leveraging a patient's prior scans to adapt a general CNN, they achieve significantly better detection of liver metastases and segmentation of brain white matter hyperintensities. This approach holds promise for more accurate and personalized medical image analysis.

What Problem Does This Paper Address?

Medical imaging often involves serial scans for patients, particularly for monitoring disease progression or treatment response. While convolutional neural networks (CNNs) have shown great success in quantifying lesions (like tumors or white matter changes) in these images, they typically rely on large, general datasets. This can lead to suboptimal performance for individual patients due to variations in image quality, anatomy, and disease presentation. The challenge is to effectively utilize the rich, patient-specific information present in their historical scans to enhance the accuracy of current lesion quantification, rather than treating each scan in isolation.

Methods

The core innovation is patient-specific fine-tuning (FT). The authors start with a "base CNN" pre-trained on a large, diverse dataset. Then, for each new patient, they take a previously acquired scan (the "baseline scan") and use it to further train (fine-tune) the last few layers of the base CNN. This process adapts the network's learned features to the specific characteristics of that patient's anatomy and lesions.

The FT process involves:

  • Dual-Pathway Architecture: The base CNN uses a dual-pathway architecture to process multiple MRI sequences simultaneously, capturing complementary information.
  • Selective Fine-Tuning: Only the last two layers of the CNN are fine-tuned. This is a computational efficiency measure that still allows the network to adapt its classification capabilities based on the patient-specific features extracted by the earlier layers.
  • Data Augmentation: Techniques like random rotation of image patches are used during FT to increase the robustness of the fine-tuned model.
  • Weighted Loss Function: For lesion detection, specific weights are assigned to false negatives (missed lesions) to encourage the network to be more sensitive. For segmentation, weights are used to balance class imbalance.

Datasets

The study utilizes two distinct datasets for evaluation:

  • Liver Metastases Detection: Abdominal MRI from 47 patients with liver metastases. Data from 31 patients with only one scan were used for training the base CNN, while 16 patients with both baseline and follow-up scans were used for testing.
  • White Matter Hyperintensities (WMH) Segmentation: Brain MRI from 80 patients with WMH. Data from 60 patients with single scans (from the WMH challenge dataset) were used for training the base CNN, and 20 patients with two scans were used for testing.

The paper does not introduce new benchmark datasets but rather uses existing clinical data for validation.

Key Insights and Ideas

  • Leveraging Historical Data is Key: The most significant insight is that a patient's own past scans contain crucial information that can dramatically improve the performance of a general CNN for current scans.
  • Patient-Specific Adaptation Outperforms Generalization: A fine-tuned CNN is demonstrably better at quantifying lesions for an individual patient than a CNN trained on a broad population.
  • Efficiency of Fine-Tuning: Fine-tuning only the last few layers is computationally efficient and effective, avoiding the need to retrain the entire network.
  • Impact on Small Lesions: Patient-specific FT appears particularly beneficial for detecting smaller lesions, which are often missed by general CNNs.
  • Reduced Uncertainty: The fine-tuned CNNs not only improve accuracy but also become more "certain" about their predictions, as indicated by lower standard deviations in their output probabilities.

Key Results

  • Liver Metastases Detection:
    • Median True Positive Rate (TPR) increased from 0.67 (base CNN) to 0.85 (patient-specific FT).
  • Median F1 score increased from 0.49 to 0.57.
  • The improvement was particularly notable for smaller metastases (<1 cm³).
  • WMH Segmentation:
    • Mean Dice Similarity Coefficient (DSC) increased from 0.82 (base CNN) to 0.87 (patient-specific FT).
  • Mean Absolute Volume Difference (AVD) decreased from 20.7% to 10.7%.
  • CNN Uncertainty: The mean standard deviation of Softmax probabilities decreased significantly for the patient-specific CNNs, indicating increased confidence in their predictions.

Limitations and Future Work

  • Overfitting Risk: While not observed in their experiments, the authors acknowledge that patient-specific FT can be prone to overfitting, especially if the baseline and follow-up scans differ significantly due to treatment or acquisition variations. They suggest limiting the number of FT iterations to mitigate this.
  • Loss Function Design: The current pixel-level loss function is more suited for segmentation than detection. They propose exploring object-level loss functions for detection tasks to potentially reduce false positives and negatives.
  • Data Imbalance: For liver metastases, an imbalanced dataset (few slices with lesions) led to challenges. They suggest careful slice selection or full lesion annotation to address this.
  • Spatial Information: The current method primarily uses appearance. Future work could incorporate spatial information from baseline scans (e.g., using image registration) to further improve detection and account for lesion development or disappearance.
  • Generalizability: While tested on two applications, the authors believe the FT approach can be applied to other network architectures and clinical settings.

Summary updated

Abstract

<b>Purpose:</b> Convolutional neural network (CNN) methods have been proposed to quantify lesions in medical imaging. Commonly, more than one imaging examination is available for a patient, but the serial information in these images often remains unused. CNN-based methods have the potential to extract valuable information from previously acquired imaging to better quantify lesions on current imaging of the same patient. <b>Approach:</b> A pretrained CNN can be updated with a patient's previously acquired imaging: patient-specific fine-tuning (FT). In this work, we studied the improvement in performance of lesion quantification methods on magnetic resonance images after FT compared to a pretrained base CNN. We applied the method to two different approaches: the detection of liver metastases and the segmentation of brain white matter hyperintensities (WMH). <b>Results:</b> The patient-specific fine-tuned CNN has a better performance than the base CNN. For the liver metastases, the median true positive rate increases from 0.67 to 0.85. For the WMH segmentation, the mean Dice similarity coefficient increases from 0.82 to 0.87. <b>Conclusions:</b> We showed that patient-specific FT has the potential to improve the lesion quantification performance of general CNNs by exploiting a patient's previously acquired imaging.
BibTeX
@article{jansen2020patient,
  author = {Mariëlle J. A. Jansen and Hugo J. Kuijf and Ashis Kumar Dhara and Nick A. Weaver and Geert Jan Biessels and Robin Strand and Josien P. W. Pluim},
  title = {{Patient-specific fine-tuning of convolutional neural networks for follow-up lesion quantification}},
  journal = {Journal of Medical Imaging},
  year = {2020},
  doi = {10.1117/1.jmi.7.6.064003},
}
Figure