Deep Mesh Projectors for Inverse Problems

Sidharth Gupta, Konik Kothari, Maarten V. de Hoop, Ivan Dokmanic

Abstract

We propose a new learning-based approach to solve ill-posed inverse problems in imaging. We address the case where ground truth training samples are rare and the problem is severely ill-posed—both because of the underlying physics and because we can only get few measurements. This setting is common in geophysical imaging and remote sensing. We show that in this case the common approach to directly learn the mapping from the measured data to the reconstruction becomes unstable. Instead, we propose to first learn an ensemble of simpler mappings from the data to projections of the unknown image into random piecewise-constant subspaces. We then combine the projections to form a final reconstruction by solving a deconvolution-like problem. We show experimentally that the proposed method is more robust to measurement noise and corruptions not seen during training than a directly learned inverse.

Introduction

In many imaging fields, such as geophysics and remote sensing, we encounter "severely ill-posed" inverse problems. These problems require reconstructing a high-resolution image $x$ from a limited set of noisy measurements $y$ via a forward operator $A$, where the number of measurements $M$ is significantly smaller than the number of pixels $N$ ($M \ll N$).

While deep learning has seen success in medical imaging, it faces significant hurdles in these domains. First, there is a severe scarcity of ground truth data; for instance, verified maps of the Earth's deep interior are rare. Second, direct neural network inversions often become unstable when measurements are sparse, frequently "hallucinating" features from the training set that are absent in the actual data, particularly under noisy conditions.

Comparison of reconstruction methods for moderate and severe ill-posed problems.

Methodology

The proposed method employs a "divide and conquer" strategy to decompose the complex reconstruction task into multiple simpler sub-tasks.

Stage 1: Learning Projections

The unknown image is projected onto multiple random Delaunay triangle meshes, each representing a subspace of piecewise-constant images. Instead of learning the full image directly, convolutional neural networks (referred to as ProjNet or SubNet) are trained to estimate the local averages (projection coefficients) for these random triangles. Mathematically, projecting into these lower-dimensional subspaces reduces the Lipschitz constant of the inverse mapping, making it "smoother" and easier to learn with limited samples.

Stage 2: Reformulated Inverse Solver

Once the local averages from various random meshes are estimated, they are combined. The authors demonstrate that, in expectation, this combination acts as a deconvolution problem. This is then solved using regularized least squares to produce the final reconstruction.

The two-stage architecture for learning projections onto random meshes.

Experimental Results

The method was evaluated using cross-domain testing to ensure it does not merely memorize training data. Models were trained on 20,000 images from the LSUN bridges dataset and evaluated on diverse benchmarks, including seismic velocity models (BP2004), real-world X-ray images (GDXray), and synthetic shapes.

Robustness and Generalization

  • Noise Stability: In tests with 10dB measurement noise, the proposed mesh-based approach consistently outperformed direct U-Net inversions. For X-ray images, it achieved a Signal-to-Noise Ratio (SNR) of ~14.5 dB, compared to ~13.5 dB for direct inversion.
  • Handling Erasures: When measurements were randomly erased, direct inversion failed to recover coarse geometry, whereas the proposed method maintained structural integrity.
  • Cross-Domain Performance: The model successfully reconstructed seismic cavities and metal parts despite being trained on bridge images, proving its robustness to the specific visual characteristics of the training set.

Discussion and Conclusion

By utilizing an ensemble of random meshes, the reconstruction process becomes effectively spatially invariant. This "convolutionalization" allows the network to learn the underlying physics of the measurements rather than specific training geometries.

While the method prioritizes stability over the sharpness often seen in direct CNN outputs, it ensures the captured geometry is physically grounded. Current limitations include the computational overhead of training multiple projection networks and a tendency toward smoother reconstructions. Future work will explore modern regularizers for the Stage 2 solver and extensions to non-linear inverse problems.

Summary updated

Abstract

We propose a new learning-based approach to solve ill-posed inverse problems in imaging. We address the case where ground truth training samples are rare and the problem is severely ill-posed - both because of the underlying physics and because we can only get few measurements. This setting is common in geophysical imaging and remote sensing. We show that in this case the common approach to directly learn the mapping from the measured data to the reconstruction becomes unstable. Instead, we propose to first learn an ensemble of simpler mappings from the data to projections of the unknown image into random piecewise-constant subspaces. We then combine the projections to form a final reconstruction by solving a deconvolution-like problem. We show experimentally that the proposed method is more robust to measurement noise and corruptions not seen during training than a directly learned inverse.
BibTeX
@inproceedings{gupta2019deep,
  author = {Sidharth Gupta and Konik Kothari and Maarten V. de Hoop and Ivan Dokmanic},
  title = {{Deep Mesh Projectors for Inverse Problems}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=Sw8YUnEblXn},
}
Figure