A Review on Deep Learning in Medical Image Reconstruction

Haimiao Zhang, Bin DongJournal of the Operations Research Society of China 2019 · DOI · OpenReview

TL;DR

This review explores the exciting intersection of deep learning and medical image reconstruction, highlighting how deep models are revolutionizing the field by offering greater flexibility and accuracy. The paper emphasizes the "unrolling dynamics" approach, which combines the interpretability of traditional mathematical models with the power of deep learning to create more effective and efficient reconstruction methods.

What Problem Does This Paper Address?

Medical imaging is indispensable for diagnosis and treatment, but acquiring high-quality images often involves trade-offs between cost, risk, and image fidelity. Image reconstruction is a critical step in this process, aiming to produce clear images from raw sensor data. Historically, this relied on "handcrafted" mathematical models based on prior knowledge. While interpretable and theoretically sound, these models struggle to fully leverage the vast amounts of data now available. Deep learning models, on the other hand, excel at extracting complex patterns from data but often lack theoretical grounding and interpretability. This paper addresses the challenge of bridging this gap by reviewing how deep learning is being integrated into medical image reconstruction, focusing on methods that combine the strengths of both handcrafted and data-driven approaches.

Methods

The paper reviews several key deep learning architectures and methodologies relevant to medical image reconstruction:

  • Handcrafted Models: These are based on mathematical principles and prior knowledge about image properties (e.g., sparsity, smoothness). Examples include Total Variation (TV) and wavelet-based methods.
  • Handcraft + Data-Driven Models: These combine handcrafted frameworks with learned components, allowing them to better utilize data while retaining some interpretability.
  • Deep Learning Models (Deep Models): These are primarily data-driven, with architectures like Convolutional Neural Networks (CNNs), Residual Networks (ResNets), Autoencoders, and U-Nets. They learn complex, nonlinear mappings from raw data or intermediate reconstructions to high-quality images.
  • Unrolling Dynamics: A central theme is the "unrolling dynamics" approach. This involves taking iterative optimization algorithms used in handcrafted models and "unrolling" them into feed-forward neural networks. This allows for end-to-end training, where parameters of the optimization steps are learned from data, leading to more effective and interpretable deep models. Examples include unrolling algorithms like ISTA and ADMM.
  • Differential Equation and Control Perspectives: The paper also discusses how deep neural networks, particularly ResNets, can be interpreted as discretizations of differential equations or as dynamic systems. This perspective offers insights into network design and training.
  • Algorithm Design: Various optimization algorithms crucial for training deep models are reviewed, including ISTA, FISTA, ADMM/Split Bregman, Primal-Dual algorithms, and Stochastic Gradient Descent (SGD).

Datasets

Not explicitly discussed in the paper. The paper focuses on the methodologies and theoretical underpinnings of deep learning in medical image reconstruction, rather than specific experimental setups with particular datasets.

Key Insights and Ideas

  • The "Unrolling Dynamics" Sweet Spot: Combining handcrafted mathematical models with deep learning through unrolling optimization algorithms offers a powerful paradigm. It allows for models that are more interpretable than black-box deep networks, often require fewer parameters (making them suitable for limited data), and effectively integrate domain knowledge.
  • Deep Models as Function Approximators: Deep neural networks are powerful function approximators, capable of learning complex, nonlinear mappings. Their depth and width play crucial roles in their approximation power.
  • Bridging Theory and Practice: The paper highlights the ongoing effort to provide theoretical foundations for deep learning models, moving beyond empirical success. Interpreting DNNs through the lens of differential equations and dynamical systems offers a path towards more principled design.
  • Task-Specific Quality Metrics: A key difference in medical imaging is the need for task-based quality metrics, rather than generic ones like PSNR or SSIM. Deep learning can be used to learn these task-specific metrics implicitly.
  • Raw-to-Image vs. Post-Processing: Two main strategies for applying deep learning in reconstruction are discussed: post-processing (enhancing already reconstructed images) and raw-to-image (directly mapping raw sensor data to the final image). The latter is often more challenging but potentially more powerful.

Key Results

As this is a review paper, it doesn't present new experimental results. Instead, it synthesizes findings from numerous existing works. The key "results" are the insights and trends identified:

  • Deep learning models have significantly advanced the state-of-the-art in various medical imaging reconstruction tasks.
  • The unrolling dynamics approach has shown promise in creating deep models that are both effective and more interpretable than traditional deep networks.
  • Interpreting deep networks through the lens of differential equations and dynamical systems provides a framework for designing better architectures and algorithms.
  • Combining image reconstruction with downstream analysis tasks (e.g., diagnosis) in an end-to-end fashion can lead to improved performance and more relevant quality metrics.

Limitations and Future Work

  • Limited Labeled Data: A significant challenge in medical imaging is the scarcity of large, well-annotated datasets. This limits the training of complex deep models. Future work should focus on designing models that can effectively utilize limited labeled data and abundant unlabeled data.
  • Interpretability and Theoretical Foundations: While progress is being made, deep models still often lack robust theoretical foundations and can be difficult to interpret. Further research is needed to understand their inner workings and ensure reliability.
  • Incorporating Diverse Data Sources: Radiologists use more than just images for diagnosis. Future models should aim to integrate information from multiple sources (e.g., patient history, clinical data) to improve decision-making.
  • Reasoning and Causal Inference: Current deep models primarily perform inference. Incorporating reasoning capabilities, such as causal inference or medical knowledge graphs, could lead to more reliable predictions and reduce the need for extensive annotated data.
  • Task-Based Quality Metrics: Developing and implementing effective task-based quality metrics for medical images remains an active area of research.

Summary updated

Abstract

Medical imaging is crucial in modern clinics to guide the diagnosis and treatment of diseases. Medical image reconstruction is one of the most fundamental and important components of medical imaging, whose major objective is to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Mathematical models in medical image reconstruction or, more generally, image restoration in computer vision, have been playing a prominent role. Earlier mathematical models are mostly designed by human knowledge or hypothesis on the image to be reconstructed, and we shall call these models handcrafted models. Later, handcrafted plus data-driven modeling started to emerge which still mostly relies on human designs, while part of the model is learned from the observed data. More recently, as more data and computation resources are made available, deep learning based models (or deep models) pushed data-driven modeling to the extreme where the models are mostly based on learning with minimal human designs. Both handcrafted and data-driven modeling have their own advantages and disadvantages. One of the major research trends in medical imaging is to combine handcrafted modeling with deep modeling so that we can enjoy benefits from both approaches. The major part of this article is to provide a conceptual review of some recent works on deep modeling from the unrolling dynamics viewpoint. This viewpoint stimulates new designs of neural network architectures with inspiration from optimization algorithms and numerical differential equations. Given the popularity of deep modeling, there are still vast remaining challenges in the field, as well as opportunities which we shall discuss at the end of this article.
BibTeX
@article{zhang2019review,
  author = {Haimiao Zhang and Bin Dong},
  title = {{A Review on Deep Learning in Medical Image Reconstruction}},
  journal = {Journal of the Operations Research Society of China},
  year = {2019},
  doi = {10.1007/s40305-019-00287-4},
}
Figure