Abstract
Unrolled neural networks have become a standard approach for solving inverse problems in imaging, yet their generalization behavior remains poorly understood. This study utilizes Stein’s Unbiased Risk Estimator (SURE) to analyze the Degrees of Freedom (DOF) in these architectures. We demonstrate that recurrent unrolled networks, which employ weight sharing across iterations, provide a strong inductive bias that significantly improves performance in data-limited regimes. Our empirical results across denoising, deblurring, and MRI reconstruction tasks show that while weight sharing is crucial when training samples are scarce, its advantage diminishes as the dataset size grows, highlighting recurrence as a powerful regularizer for label-constrained scenarios.
Introduction
In medical imaging domains such as MRI and CT, acquiring high-quality ground truth labels for training deep learning models is often impractical due to physical constraints and patient privacy. To address these inverse problems—recovering an underlying image from noisy or undersampled measurements—researchers frequently employ unrolled neural networks. These models are structured to mimic a fixed number of iterations of classical optimization algorithms, such as Gradient Descent, but replace traditional priors with learned components.
Despite their empirical success, a rigorous mathematical understanding of how these networks generalize based on training data availability has been lacking. A key question is whether sharing weights across the unrolled iterations (recurrent architectures) offers a fundamental advantage over using unique weights for each iteration (non-recurrent architectures), particularly when training data is limited.
Theoretical Framework: SURE and Degrees of Freedom
To quantify generalization risk without relying on ground truth labels, we leverage Stein’s Unbiased Risk Estimator (SURE). SURE provides an estimate of the Mean Squared Error (MSE) by decomposing the risk into two components:
- Residual Sum of Squares (RSS): A measure of the model's prediction bias.
- Degrees of Freedom (DOF): A measure of prediction variance or model complexity, calculated as the trace of the end-to-end network Jacobian.
Our analysis shows that the DOF of unrolled networks is well-approximated by their weighted path sparsity. We compare two primary training schemes: Weight Sharing (WS), a recurrent approach where parameters are tied across all iterations, and Weight Changing (WC), where each iteration possesses independent parameters.
Experimental Results
Natural Image Denoising
We evaluated WS and WC schemes on the BSD68 dataset for denoising tasks across various noise levels ($\sigma$). In the low-sample regime (fewer than 1,000 training samples), the recurrent WS architecture consistently outperformed the non-recurrent WC architecture in terms of Peak Signal-to-Noise Ratio (PSNR).
As shown in the figure above, the performance gap is most pronounced when the sample size is small. Interestingly, as the training set size increases beyond $10^5$ samples, the performance of both architectures converges. This suggests that the architectural constraint of recurrence acts as a form of regularization that is specifically beneficial when labels are scarce.
Natural Image Deblurring
Similarly, in deblurring tasks using Gaussian kernels, the WS scheme maintains a performance lead in low-data scenarios.
The results indicate that the recurrent model reaches saturation in complexity faster than the non-recurrent model, allowing it to learn the underlying task structure with significantly fewer examples.
Compressed Sensing MRI
The benefits of recurrence were further validated on a clinical MRI dataset consisting of 19 subjects. When trained on a very small number of MR images, the weight-sharing version achieved a ~1 dB gain in PSNR over the weight-changing version for reconstructing images from 20% Fourier coefficients.
Key Insights
- Recurrence as a Data-Saver: Weight sharing reduces the effective degrees of freedom, allowing the network to saturate its learning capacity with fewer samples.
- Filtering Dynamics: Analysis of learned filters revealed that recurrent networks consistently perform bandpass filtering. In contrast, non-recurrent networks alternate between lowpass and bandpass filtering across iterations to achieve denoising.
- SURE as a Monitoring Tool: Empirical results confirm that SURE-based estimates, which do not require ground truth, accurately track test MSE, making it a viable tool for real-world training monitoring in label-scarce environments.
Conclusion
This work provides a theoretical foundation for the use of recurrent unrolled networks in inverse problems. By analyzing the degrees of freedom, we show that weight sharing is a highly effective regularization strategy for overcoming label scarcity. Future research will focus on extending this SURE analysis to arbitrary sensing matrices and exploring the interaction between weight sharing and other regularization techniques like early stopping.
Summary updated