Diff-Unfolding: A Model-Based Score Learning Framework for Inverse Problems

Yuanhao Wang, Shirin Shoushtari, Ulugbek S. KamilovarXiv 2025 · DOI · arXiv

Abstract

Diffusion models are extensively used for modeling image priors for inverse problems. We introduce Diff-Unfolding, a principled framework for learning posterior score functions of conditional diffusion models by explicitly incorporating the physical measurement operator into a modular network architecture. Diff-Unfolding formulates posterior score learning as the training of an unrolled optimization scheme, where the measurement model is decoupled from the learned image prior. This design allows our method to generalize across inverse problems at inference time by simply replacing the forward operator without retraining. Extensive experiments on image restoration and accelerated MRI show that Diff-Unfolding achieves state-of-the-art performance, improving PSNR by up to 2 dB and reducing LPIPS by 22.7%, while being both compact (47M parameters) and efficient (0.72 seconds per 256x256 image). An optimized C++/LibTorch implementation further reduces inference time to 0.63 seconds, underscoring the practicality of our approach.

Introduction

Solving "inverse problems"—recovering a clean image from noisy, blurry, or incomplete measurements—is a cornerstone of medical imaging and computer vision. Traditionally, researchers used handcrafted mathematical priors. Recently, Diffusion Models (DMs) have emerged as powerful alternatives because they can generate complex textures far better than classical mathematical models.

However, current DM-based solvers face a significant trade-off. Unconditional DMs (UDMs) use general-purpose pre-trained models and "guide" them during inference; while flexible, they often rely on mathematical approximations that lead to artifacts. Conversely, Conditional DMs (CDMs) are trained for specific tasks, producing high-quality results but "entangling" the image prior with the specific degradation. If the measurement process changes even slightly, the entire model usually requires expensive retraining.

Diff-Unfolding provides a "best of both worlds" approach: the high-fidelity performance of conditional models with the "plug-and-play" flexibility of unconditional ones.

Methods

The core innovation of Diff-Unfolding is the integration of the physical measurement operator directly into the neural network's architecture using Deep Unfolding (DU).

Overview of the Diff-Unfolding framework showing the unrolled sampling process and various image restoration tasks like deblurring, inpainting, and super-resolution.

The Unrolling Strategy

Instead of treating the network as a "black box," the authors formulate the estimation of the posterior score (the direction to move to get a cleaner image) as an unrolled optimization algorithm. Each "layer" of the network represents one iteration of a gradient-descent-like scheme. Within these layers, the physical measurement model (the "data fidelity" term) is kept separate from the learned image prior (the "denoiser").

Theoretical Grounding and Adaptive Weighting

The framework is mathematically justified by showing that the optimal denoiser for a conditional diffusion model is the solution to a composite optimization problem. To handle varying noise levels during the diffusion process, the framework uses small auxiliary MLP networks to learn how to balance the prior and the measurements. At high noise levels, the model trusts the learned prior more; as the image clears, it shifts trust toward the raw measurement data.

Algorithms for training and sampling in the Diff-Unfolding framework, illustrating the iterative unrolling process.

Key Insights

  • Architecture as a Prior: By forcing the network to follow the structure of an optimization algorithm, the model inherently understands the physics of the measurement, making it more interpretable and robust.
  • Zero-Shot Generalization: Because the forward operator is decoupled, a model trained on one type of blur can, at inference time, be applied to a different blur kernel or a super-resolution operator without retraining.
  • Efficiency via Compactness: The modular design achieves state-of-the-art results with only 47M parameters, significantly fewer than standard conditional diffusion models that often exceed 100M parameters.

Experimental Results

The researchers evaluated Diff-Unfolding on the FFHQ face dataset and the fastMRI Multicoil Knee dataset.

  • Superior Quality: Diff-Unfolding outperformed existing state-of-the-art methods like DPS and cEDM, improving Peak Signal-to-Noise Ratio (PSNR) by up to 2 dB and reducing perceptual error (LPIPS) by 22.7%.
  • Speed: The model is remarkably fast, reconstructing a single image in 0.72 seconds (0.63 seconds with C++ optimization), whereas some diffusion solvers take minutes.
  • MRI Performance: In MRI tasks, the method significantly reduced aliasing artifacts compared to traditional score-approximation methods, accurately preserving fine anatomical structures like cartilage.

Limitations and Future Work

While powerful, the current implementation focuses on linear inverse problems where degradation is expressed as matrix multiplication. Extending this to non-linear problems, such as phase retrieval, is a primary goal for future research. Additionally, while the model generalizes well, extremely different forward operators might still benefit from light fine-tuning to reach peak performance. Future iterations may also adapt the framework for more complex noise profiles, such as Poisson noise, beyond the current additive white Gaussian noise optimization.

Summary updated

Abstract

Diffusion models are extensively used for modeling image priors for inverse problems. We introduce \emph{Diff-Unfolding}, a principled framework for learning posterior score functions of \emph{conditional diffusion models} by explicitly incorporating the physical measurement operator into a modular network architecture. Diff-Unfolding formulates posterior score learning as the training of an unrolled optimization scheme, where the measurement model is decoupled from the learned image prior. This design allows our method to generalize across inverse problems at inference time by simply replacing the forward operator without retraining. We theoretically justify our unrolling approach by showing that the posterior score can be derived from a composite model-based optimization formulation. Extensive experiments on image restoration and accelerated MRI show that Diff-Unfolding achieves state-of-the-art performance, improving PSNR by up to 2 dB and reducing LPIPS by $22.7\%$, while being both compact (47M parameters) and efficient (0.72 seconds per $256 \times 256$ image). An optimized C++/LibTorch implementation further reduces inference time to 0.63 seconds, underscoring the practicality of our approach.
BibTeX
@misc{wang2025diff,
  author = {Yuanhao Wang and Shirin Shoushtari and Ulugbek S. Kamilov},
  title = {{Diff-Unfolding: A Model-Based Score Learning Framework for Inverse Problems}},
  year = {2025},
  eprint = {2505.11393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.11393},
  howpublished = {arXiv},
}
Figure