Efficient Photoacoustic Image Synthesis with Deep Learning

Tom Rix, Kris K. Dreher, Jan-Hinrich Nölke, Melanie Schellenberg, Minu Dietlinde Tizabi, Alexander Seitel, Lena Maier-Hein

TL;DR

This paper introduces a deep learning-based approach to efficiently synthesize photoacoustic (PA) images, overcoming the computational bottleneck of traditional Monte Carlo (MC) simulations. By training neural networks (U-Net and Fourier Neural Operator) on synthetic data, they achieve a 100x speedup in image generation while maintaining comparable accuracy to MC simulations with significantly fewer photons, paving the way for faster clinical applications of PA imaging.

What Problem Does This Paper Address?

Photoacoustic imaging (PAI) holds immense promise for non-invasively visualizing functional tissue properties like oxygenation. However, accurately quantifying these properties from PAI signals is a complex, ill-posed problem. While deep learning (DL) has shown potential for solving this, a major hurdle is the lack of efficient and accurate methods for generating the vast amounts of synthetic data needed for training DL models. Current state-of-the-art methods for simulating the optical part of PAI, specifically photon propagation through tissue, rely on Monte Carlo (MC) simulations. These are computationally very expensive and, crucially, are not inherently differentiable, preventing their integration into gradient-based optimization pipelines for improving image synthesis realism. This paper aims to bridge this gap by developing a fast, differentiable, deep learning-based surrogate model for simulating photon propagation in tissue, enabling efficient photoacoustic image synthesis.

Methods

The core of this work is the development of deep neural network architectures to replace computationally intensive MC simulations for the optical forward step in PAI image synthesis. The authors investigate two complementary network architectures:

  • U-Net: A well-established convolutional neural network architecture commonly used for image-to-image tasks. They adapted it with three input channels and a single output channel, employing residual skip connections for combining local and global features.
  • Fourier Neural Operator (FNO): This network operates in Fourier space, allowing it to capture global context and learn sinusoidal features, making it well-suited for approximating continuous physical functions like photon propagation. It involves projecting data into a latent space, applying Fourier layers with spectral convolutions, and then a 2D convolution.

Both networks are trained in a supervised manner using pairs of optical parameter maps (absorption, scattering coefficient, and anisotropy) and the corresponding initial pressure distributions generated by MC simulations. The initial pressure distributions are logarithmically transformed for training. The key innovation is that these DL models are designed to be differentiable, allowing for back-propagation of errors and gradient-based optimization of the entire PAI synthesis pipeline.

Datasets

The authors generated a synthetic dataset of human forearm photoacoustic data. This dataset was created using the SIMPA toolkit, incorporating a literature-based anatomy model of the human forearm, including radial and ulnar arteries and veins. Optical parameters (absorption coefficient, scattering coefficient, and scattering anisotropy) were assigned based on literature values for 16 wavelengths (700-850 nm). The simulations mimicked a specific MSOT Acuity Echo device. The dataset consists of 1100 samples, each with 16 wavelengths. This data was split into training (770 samples), validation (110 samples), and test (220 samples) sets.

Key Insights and Ideas

  • DL as a Differentiable Surrogate: The most significant insight is that deep learning models can serve as differentiable surrogate models for photon propagation. This is crucial because it allows for end-to-end optimization of the entire PAI synthesis pipeline, not just the forward simulation step.
  • FNO vs. U-Net Performance: While both architectures perform well, the FNO consistently shows slightly better accuracy, likely due to its ability to capture global features more effectively in Fourier space.
  • Speed vs. Accuracy Trade-off: The DL models achieve a dramatic speedup (100x) compared to MC simulations. Importantly, this speedup is achieved while maintaining accuracy comparable to MC simulations that used 100 times fewer photons. This suggests that DL can effectively "learn" the essential physics without needing the full stochastic detail of high-photon-count MC.
  • Error Nature Difference: A key qualitative finding is that DL errors tend to occur at structure boundaries, whereas MC errors with fewer photons manifest as general noise. This difference is important for understanding the limitations and potential applications of each method.

Key Results

  • Speedup: The DL models (U-Net and FNO) achieve inference run times that are two orders of magnitude faster than MC simulations, completing the optical forward step in under a second compared to minutes for MC.
  • Accuracy: The DL models achieve accuracy comparable to MC simulations using 100 times fewer photons (5x10^6 vs. 5x10^8 photons). The FNO slightly outperforms the U-Net in terms of mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM).
  • Training Data Impact: Increasing the amount of training data improved performance up to a point, after which it saturated.
  • Qualitative Error Analysis: Misestimations in DL models are observed in high-absorbing structures like vessels and skin, particularly in samples with thicker skin or prominent vessels.

Limitations and Future Work

  • Generalizability: The current models are trained on specific forearm anatomy and a single device configuration. Retraining is required for different body sites, optical parameter ranges, or device geometries, limiting immediate applicability. Future work could focus on developing more generic models or sharing trained models.
  • Out-of-Distribution Cases: The study was limited to in-distribution data. Investigating extrapolation beyond training wavelengths and interpolation between them is suggested.
  • Model Accuracy Limit: The accuracy of the DL surrogate is inherently limited by the accuracy of the MC simulations used for training.
  • Clinical Validation: The authors suggest future work should validate the synthesis results in a downstream task to demonstrate equivalence to state-of-the-art methods and explore the benefits of differentiability over non-differentiable models.
  • Physics Integration: Incorporating more explicit physics prior knowledge into the DL architecture or approach could lead to even more accurate results and faster convergence.

Summary updated

Abstract

Photoacoustic imaging potentially allows for the real-time visualization of functional human tissue parameters such as oxygenation but is subject to a challenging underlying quantification problem. While in silico studies have revealed the great potential of deep learning (DL) methodology in solving this problem, the inherent lack of an efficient gold standard method for model training and validation remains a grand challenge. This work investigates whether DL can be leveraged to accurately and efficiently simulate photon propagation in biological tissue, enabling photoacoustic image synthesis. Our approach is based on estimating the initial pressure distribution of the photoacoustic waves from the underlying optical properties using a back-propagatable neural network trained on synthetic data. In proof-of-concept studies, we validated the performance of two complementary neural network architectures, namely a conventional U-Net-like model and a Fourier Neural Operator (FNO) network. Our in silico validation on multispectral human forearm images shows that DL methods can speed up image generation by a factor of 100 when compared to Monte Carlo simulations with 5 × 10 8 photons. While the FNO is slightly more accurate than the U-Net, when compared to Monte Carlo simulations performed with a reduced number of photons ( 5 × 10 6 ), both neural network architectures achieve equivalent accuracy. In contrast to Monte Carlo simulations, the proposed DL models can be used as inherently differentiable surrogate models in the photoacoustic image synthesis pipeline, allowing for back-propagation of the synthesis error and gradient-based optimization over the entire pipeline. Due to their efficiency, they have the potential to enable large-scale training data generation that can expedite the clinical application of photoacoustic imaging.
BibTeX
@inproceedings{rix2023efficient,
  author = {Tom Rix and Kris K. Dreher and Jan-Hinrich Nölke and Melanie Schellenberg and Minu Dietlinde Tizabi and Alexander Seitel and Lena Maier-Hein},
  title = {{Efficient Photoacoustic Image Synthesis with Deep Learning}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=wIfxNuVv4v},
}

Loading related content...

Figure