Abstract
Diffusion models are extensively used for modeling image priors for inverse problems. We introduce Diff-Unfolding, a principled framework for learning posterior score functions of conditional diffusion models by explicitly incorporating the physical measurement operator into a modular network architecture. Diff-Unfolding formulates posterior score learning as the training of an unrolled optimization scheme, where the measurement model is decoupled from the learned image prior. This design allows our method to generalize across inverse problems at inference time by simply replacing the forward operator without retraining. Extensive experiments on image restoration and accelerated MRI show that Diff-Unfolding achieves state-of-the-art performance, improving PSNR by up to 2 dB and reducing LPIPS by 22.7%, while being both compact (47M parameters) and efficient (0.72 seconds per 256x256 image). An optimized C++/LibTorch implementation further reduces inference time to 0.63 seconds, underscoring the practicality of our approach.
Introduction
Solving "inverse problems"—recovering a clean image from noisy, blurry, or incomplete measurements—is a cornerstone of medical imaging and computer vision. Traditionally, researchers used handcrafted mathematical priors. Recently, Diffusion Models (DMs) have emerged as powerful alternatives because they can generate complex textures far better than classical mathematical models.
However, current DM-based solvers face a significant trade-off. Unconditional DMs (UDMs) use general-purpose pre-trained models and "guide" them during inference; while flexible, they often rely on mathematical approximations that lead to artifacts. Conversely, Conditional DMs (CDMs) are trained for specific tasks, producing high-quality results but "entangling" the image prior with the specific degradation. If the measurement process changes even slightly, the entire model usually requires expensive retraining.
Diff-Unfolding provides a "best of both worlds" approach: the high-fidelity performance of conditional models with the "plug-and-play" flexibility of unconditional ones.
Methods
The core innovation of Diff-Unfolding is the integration of the physical measurement operator directly into the neural network's architecture using Deep Unfolding (DU).
The Unrolling Strategy
Instead of treating the network as a "black box," the authors formulate the estimation of the posterior score (the direction to move to get a cleaner image) as an unrolled optimization algorithm. Each "layer" of the network represents one iteration of a gradient-descent-like scheme. Within these layers, the physical measurement model (the "data fidelity" term) is kept separate from the learned image prior (the "denoiser").
Theoretical Grounding and Adaptive Weighting
The framework is mathematically justified by showing that the optimal denoiser for a conditional diffusion model is the solution to a composite optimization problem. To handle varying noise levels during the diffusion process, the framework uses small auxiliary MLP networks to learn how to balance the prior and the measurements. At high noise levels, the model trusts the learned prior more; as the image clears, it shifts trust toward the raw measurement data.
Key Insights
- Architecture as a Prior: By forcing the network to follow the structure of an optimization algorithm, the model inherently understands the physics of the measurement, making it more interpretable and robust.
- Zero-Shot Generalization: Because the forward operator is decoupled, a model trained on one type of blur can, at inference time, be applied to a different blur kernel or a super-resolution operator without retraining.
- Efficiency via Compactness: The modular design achieves state-of-the-art results with only 47M parameters, significantly fewer than standard conditional diffusion models that often exceed 100M parameters.
Experimental Results
The researchers evaluated Diff-Unfolding on the FFHQ face dataset and the fastMRI Multicoil Knee dataset.
- Superior Quality: Diff-Unfolding outperformed existing state-of-the-art methods like DPS and cEDM, improving Peak Signal-to-Noise Ratio (PSNR) by up to 2 dB and reducing perceptual error (LPIPS) by 22.7%.
- Speed: The model is remarkably fast, reconstructing a single image in 0.72 seconds (0.63 seconds with C++ optimization), whereas some diffusion solvers take minutes.
- MRI Performance: In MRI tasks, the method significantly reduced aliasing artifacts compared to traditional score-approximation methods, accurately preserving fine anatomical structures like cartilage.
Limitations and Future Work
While powerful, the current implementation focuses on linear inverse problems where degradation is expressed as matrix multiplication. Extending this to non-linear problems, such as phase retrieval, is a primary goal for future research. Additionally, while the model generalizes well, extremely different forward operators might still benefit from light fine-tuning to reach peak performance. Future iterations may also adapt the framework for more complex noise profiles, such as Poisson noise, beyond the current additive white Gaussian noise optimization.
Summary updated