Abstract
Many important problems in science and engineering involve inferring a signal from noisy and/or incomplete observations, where the observation process is known. Historically, this problem has been tackled using hand-crafted regularization (e.g., sparsity, total-variation) to obtain meaningful estimates. Recent data-driven methods often offer better solutions by directly learning a solver from examples of ground-truth signals and associated observations. However, in many real-world applications, obtaining ground-truth references for training is expensive or impossible. Self-supervised learning (SSL) methods offer a promising alternative by learning a solver from measurement data alone, bypassing the need for ground-truth references. This manuscript provides a comprehensive summary of different self-supervised methods for inverse problems, with a special emphasis on their theoretical underpinnings, and presents practical applications in imaging inverse problems.
Introduction
In fields ranging from medical imaging (MRI/CT) to astronomy and microscopy, researchers face a fundamental "inverse problem": recovering a clean, high-fidelity signal ($x$) from noisy, blurry, or incomplete measurements ($y$). While deep learning has revolutionized this domain, standard supervised learning paradigms rely on massive datasets of "clean" ground-truth images paired with their degraded counterparts.
In many critical real-world applications, obtaining these clean references is physically impossible or ethically challenging. For instance, obtaining a "perfect" MRI scan might require a patient to remain motionless for hours, which is impractical. Furthermore, supervised models often suffer from poor generalization when encountering distribution shifts between training and test data. This monograph explores the theoretical and practical roadmap for training deep learning models to solve these problems using only the raw, imperfect measurements collected in the field.
Methodological Frameworks for Self-Supervision
The paper categorizes SSL methods based on the specific information available during the training process:
1. Learning from Noisy Pairs (Noise2Noise)
When two independent noisy measurements of the same underlying object are available, a network can be trained to map one to the other. Mathematically, the network learns the conditional mean of the distribution. Because the noise is assumed to be zero-mean and independent between the two samples, the network effectively learns to remove the noise without ever seeing a clean image.
2. Bootstrapping from Single Measurements (R2R & SURE)
In scenarios where only a single noisy measurement per object is available, researchers can employ statistical bootstrapping techniques:
- Recorrupted2Recorrupted (R2R): This method involves adding synthetic noise to the existing noisy measurement to create a new pair. The network is then trained to recover the original noisy measurement from the "recorrupted" version.
- Stein’s Unbiased Risk Estimator (SURE): This classical statistical tool allows for the estimation of the true error (MSE) relative to the unknown ground truth by using the network's own derivatives (the Jacobian). A key theoretical insight of this work is the demonstration that R2R and SURE are mathematically equivalent in certain limits.
3. Blind-Spot Networks
Specifically designed for denoising, these architectures utilize a spatial constraint: they prevent a pixel from "seeing itself" during the convolution process. This forces the network to predict a pixel's value based solely on its surrounding context, preventing it from simply learning the identity mapping of the noise.
4. Equivariant Imaging (EI)
For problems involving incomplete data (such as missing pixels or undersampled frequencies), Equivariant Imaging leverages physical symmetries. By assuming that the underlying signal distribution is invariant to certain transformations—such as rotations or translations—the model can learn to "fill in" the nullspace (the information the measurement process missed) by observing how the signal behaves under these transformations.
Key Theoretical Insights
The monograph highlights several critical concepts that govern the success of self-supervised inverse problem solvers:
- The Expressivity-Robustness Trade-off: There is a fundamental balance between a model's capacity and its susceptibility to noise. If the noise distribution is unknown, researchers must use "less expressive" estimators (such as Zero-Expected Divergence) to prevent the model from overfitting to the noise.
- Physics as a Supervisor: The forward model ($A$), which describes the physical measurement process, acts as a natural constraint. By ensuring that the reconstructed image remains consistent with the physical measurements, the physics itself serves as a loss function.
- The Blessing of Dimensionality: While high-dimensional data is often difficult to model, the local structure of images allows for highly accurate loss estimations (like SURE) using only a single Monte Carlo sample, making these methods computationally feasible.
Empirical Results and Performance
Validation across benchmarks like MNIST, Urban100, and fastMRI reveals that self-supervised methods are closing the gap with supervised learning:
- The $1/\sqrt{N}$ Gap: Theoretical analysis shows that the performance difference between SSL and supervised models shrinks at a rate proportional to $\sigma^2/\sqrt{N}$, where $N$ is the number of training samples.
- Performance Parity: In many standard denoising tasks, SSL methods like SURE or R2R achieve Peak Signal-to-Noise Ratio (PSNR) values within 0.1–0.2 dB of fully supervised models.
- Robustness: New variants like UNSURE (Unknown-noise SURE) demonstrate significantly higher robustness than standard SURE when the exact noise variance of the sensor is unknown or misestimated.
Limitations and Future Directions
Despite the progress in linear inverse problems, several frontiers remain. Current theoretical proofs are largely built for linear operators; extending these to non-linear physics, such as phase retrieval or inverse scattering, is a major open challenge. Additionally, training these models is memory-intensive, requiring backpropagation through complex physical operators. Future research must also address uncertainty quantification, providing reliable confidence intervals for reconstructions in high-stakes fields like medical diagnostics.
Summary updated