Learning The Invisible: A Hybrid Deep Learning-Shearlet Framework for Limited Angle Computed Tomography

Tatiana A. Bubba, Gitta Kutyniok, Matti Lassas, Maximilian März, Wojciech Samek, Samuli Siltanen, Vignesh SrinivasanarXiv 2018 · DOI · OpenReview

Abstract

The high complexity of various inverse problems poses a significant challenge to model-based reconstruction schemes, which in such situations often reach their limits. At the same time, we witness an exceptional success of data-based methodologies such as deep learning. However, in the context of inverse problems, deep neural networks mostly act as black box routines, used for instance for a somewhat unspecified removal of artifacts in classical image reconstructions. In this paper, we focus on the severely ill-posed inverse problem of limited angle computed tomography, in which entire boundary sections are not captured in the measurements. We develop a hybrid reconstruction framework that fuses model-based sparse regularization with data-driven deep learning. Our method is reliable in the sense that we only learn the part that can provably not be handled by model-based methods, while applying the theoretically controllable sparse regularization technique to the remaining parts. Such a decomposition into visible and invisible segments is achieved by means of the shearlet transform that allows to resolve wavefront sets in the phase space. Furthermore, this split enables us to assign the clear task of inferring unknown shearlet coefficients to the neural network and thereby offering an interpretation of its performance in the context of limited angle computed tomography. Our numerical experiments show that our algorithm significantly surpasses both pure model- and more data-based reconstruction methods.

The Challenge of Limited Angle CT

Computed Tomography (CT) is a cornerstone of modern medical imaging, yet it faces significant hurdles when full rotational data is unavailable. In scenarios such as dental surgery, breast tomosynthesis, or industrial non-destructive testing, the scanner may only capture a limited arc of projections (e.g., 60 to 120 degrees). This restriction creates a "missing wedge" in the Fourier domain, leading to reconstructions plagued by severe streaking artifacts and blurred edges.

Geometry of limited angle CT scanning, showing the object $f(x_1, x_2)$ and the projection line $L(\theta, s)$ at angle $\theta$.

Mathematically, this is described by the "Visibility Principle" in microlocal analysis: an edge is only "visible" if it is tangent to the scanned X-ray beams. Edges falling within the missing wedge are "invisible"—the information is simply not present in the raw data. Traditional algorithms often fail to recover these features, while standard deep learning approaches frequently act as "black boxes," potentially hallucinating details or failing to generalize.

The LtI Framework: Learning the Invisible

To address these limitations, the authors introduce Learning the Invisible (LtI), a hybrid framework that combines shearlet-based sparse regularization with deep learning. The pipeline consists of three primary steps:

  1. Recover the Visible: Using the shearlet transform—a mathematical tool superior to wavelets for representing anisotropic features like edges—the method solves an $\ell^1$-minimization problem. This reconstructs the "visible" parts of the image, providing a clean, artifact-reduced baseline containing all information present in the measurements.
  2. Learn the Invisible: The "invisible" shearlet coefficients (those corresponding to the missing wedge) are identified. A custom Convolutional Neural Network (CNN), PhantomNet, is then tasked with "inpainting" these missing coefficients based on the patterns observed in the visible ones.
  3. Fusion: The mathematically derived visible coefficients and the AI-inferred invisible coefficients are combined and transformed back into the image domain.

Microlocal Analysis and the Visible/Invisible Split

The core innovation of LtI lies in its use of microlocal analysis to partition the image features. By mapping the missing wedge of the scanner directly onto specific shearlet coefficients in the phase space, the AI is given a specific, mathematically defined task.

Illustration of the wavefront set (a) and its representation in the phase space $(x, \theta)$ (b), which allows for the separation of visible and invisible features.

This approach ensures that the neural network only attempts to fill in what is provably missing, rather than modifying the data that was actually measured. This split enhances both the interpretability and the reliability of the reconstruction.

Results and Generalization

The LtI framework demonstrates superior performance over traditional model-based methods (like Total Variation) and pure deep learning post-processing. In a 50-degree missing wedge test on synthetic ellipses, LtI achieved a Peak Signal-to-Noise Ratio (PSNR) of 36.96, significantly higher than standard CNN post-processing (30.20).

Qualitatively, LtI is uniquely capable of recovering rounded shapes of organs and bones that other methods flatten or erase. Furthermore, the framework shows remarkable generalization. A network trained on human abdomen scans successfully reconstructed a Lotus Root—a completely different structure—by relying on the mathematically sound "visible" coefficients as a foundation.

Conclusion

By fusing the theoretical rigor of shearlet-based regularization with the predictive power of deep learning, the LtI framework offers a robust solution to the limited angle CT problem. While the initial $\ell^1$-minimization is computationally intensive, the resulting reconstructions are more accurate and trustworthy than those produced by black-box AI models. Future work will focus on optimizing solver speeds and extending this hybrid logic to other incomplete-data inverse problems.

Summary updated

Abstract

The high complexity of various inverse problems poses a significant challenge to model-based reconstruction schemes, which in such situations often reach their limits. At the same time, we witness an exceptional success of data-based methodologies such as deep learning. However, in the context of inverse problems, deep neural networks mostly act as black box routines, used for instance for a somewhat unspecified removal of artifacts in classical image reconstructions. In this paper, we will focus on the severely ill-posed inverse problem of limited angle computed tomography, in which entire boundary sections are not captured in the measurements. We will develop a hybrid reconstruction framework that fuses model-based sparse regularization with data-driven deep learning. Our method is reliable in the sense that we only learn the part that can provably not be handled by model-based methods, while applying the theoretically controllable sparse regularization technique to the remaining parts. Such a decomposition into visible and invisible segments is achieved by means of the shearlet transform that allows to resolve wavefront sets in the phase space. Furthermore, this split enables us to assign the clear task of inferring unknown shearlet coefficients to the neural network and thereby offering an interpretation of its performance in the context of limited angle computed tomography. Our numerical experiments show that our algorithm significantly surpasses both pure model- and more data-based reconstruction methods.
BibTeX
@misc{bubba2018learning,
  author = {Tatiana A. Bubba and Gitta Kutyniok and Matti Lassas and Maximilian März and Wojciech Samek and Samuli Siltanen and Vignesh Srinivasan},
  title = {{Learning The Invisible: A Hybrid Deep Learning-Shearlet Framework for Limited Angle Computed Tomography}},
  year = {2018},
  eprint = {1811.04602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.04602},
}
Figure