Degrees of Freedom Analysis of Unrolled Neural Networks

Morteza Mardani, Qingyun Sun, Vardan Papyan, Shreyas Vasanawala, John M. Pauly, David L. DonohoarXiv 2019 · arXiv

Abstract

Unrolled neural networks have become a standard approach for solving inverse problems in imaging, yet their generalization behavior remains poorly understood. This study utilizes Stein’s Unbiased Risk Estimator (SURE) to analyze the Degrees of Freedom (DOF) in these architectures. We demonstrate that recurrent unrolled networks, which employ weight sharing across iterations, provide a strong inductive bias that significantly improves performance in data-limited regimes. Our empirical results across denoising, deblurring, and MRI reconstruction tasks show that while weight sharing is crucial when training samples are scarce, its advantage diminishes as the dataset size grows, highlighting recurrence as a powerful regularizer for label-constrained scenarios.

Introduction

In medical imaging domains such as MRI and CT, acquiring high-quality ground truth labels for training deep learning models is often impractical due to physical constraints and patient privacy. To address these inverse problems—recovering an underlying image from noisy or undersampled measurements—researchers frequently employ unrolled neural networks. These models are structured to mimic a fixed number of iterations of classical optimization algorithms, such as Gradient Descent, but replace traditional priors with learned components.

Despite their empirical success, a rigorous mathematical understanding of how these networks generalize based on training data availability has been lacking. A key question is whether sharing weights across the unrolled iterations (recurrent architectures) offers a fundamental advantage over using unique weights for each iteration (non-recurrent architectures), particularly when training data is limited.

Theoretical Framework: SURE and Degrees of Freedom

To quantify generalization risk without relying on ground truth labels, we leverage Stein’s Unbiased Risk Estimator (SURE). SURE provides an estimate of the Mean Squared Error (MSE) by decomposing the risk into two components:

  1. Residual Sum of Squares (RSS): A measure of the model's prediction bias.
  2. Degrees of Freedom (DOF): A measure of prediction variance or model complexity, calculated as the trace of the end-to-end network Jacobian.

Our analysis shows that the DOF of unrolled networks is well-approximated by their weighted path sparsity. We compare two primary training schemes: Weight Sharing (WS), a recurrent approach where parameters are tied across all iterations, and Weight Changing (WC), where each iteration possesses independent parameters.

Experimental Results

Natural Image Denoising

We evaluated WS and WC schemes on the BSD68 dataset for denoising tasks across various noise levels ($\sigma$). In the low-sample regime (fewer than 1,000 training samples), the recurrent WS architecture consistently outperformed the non-recurrent WC architecture in terms of Peak Signal-to-Noise Ratio (PSNR).

Denoising performance (PSNR) vs. sample size for WS and WC models.

As shown in the figure above, the performance gap is most pronounced when the sample size is small. Interestingly, as the training set size increases beyond $10^5$ samples, the performance of both architectures converges. This suggests that the architectural constraint of recurrence acts as a form of regularization that is specifically beneficial when labels are scarce.

Natural Image Deblurring

Similarly, in deblurring tasks using Gaussian kernels, the WS scheme maintains a performance lead in low-data scenarios.

Deblurring performance (PSNR) vs. sample size.

The results indicate that the recurrent model reaches saturation in complexity faster than the non-recurrent model, allowing it to learn the underlying task structure with significantly fewer examples.

Compressed Sensing MRI

The benefits of recurrence were further validated on a clinical MRI dataset consisting of 19 subjects. When trained on a very small number of MR images, the weight-sharing version achieved a ~1 dB gain in PSNR over the weight-changing version for reconstructing images from 20% Fourier coefficients.

Key Insights

  • Recurrence as a Data-Saver: Weight sharing reduces the effective degrees of freedom, allowing the network to saturate its learning capacity with fewer samples.
  • Filtering Dynamics: Analysis of learned filters revealed that recurrent networks consistently perform bandpass filtering. In contrast, non-recurrent networks alternate between lowpass and bandpass filtering across iterations to achieve denoising.
  • SURE as a Monitoring Tool: Empirical results confirm that SURE-based estimates, which do not require ground truth, accurately track test MSE, making it a viable tool for real-world training monitoring in label-scarce environments.

Conclusion

This work provides a theoretical foundation for the use of recurrent unrolled networks in inverse problems. By analyzing the degrees of freedom, we show that weight sharing is a highly effective regularization strategy for overcoming label scarcity. Future research will focus on extending this SURE analysis to arbitrary sensing matrices and exploring the interaction between weight sharing and other regularization techniques like early stopping.

Summary updated

Abstract

Unrolled neural networks emerged recently as an effective model for learning inverse maps appearing in image restoration tasks. However, their generalization risk (i.e., test mean-squared-error) and its link to network design and train sample size remains mysterious. Leveraging the Stein's Unbiased Risk Estimator (SURE), this paper analyzes the generalization risk with its bias and variance components for recurrent unrolled networks. We particularly investigate the degrees-of-freedom (DOF) component of SURE, trace of the end-to-end network Jacobian, to quantify the prediction variance. We prove that DOF is well-approximated by the weighted \textit{path sparsity} of the network under incoherence conditions on the trained weights. Empirically, we examine the SURE components as a function of train sample size for both recurrent and non-recurrent (with many more parameters) unrolled networks. Our key observations indicate that: 1) DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes; 2) recurrent network converges significantly faster (with less train samples) compared with non-recurrent scheme, hence recurrence serves as a regularization for low sample size regimes.
BibTeX
@misc{mardani2019degrees,
  author = {Morteza Mardani and Qingyun Sun and Vardan Papyan and Shreyas Vasanawala and John M. Pauly and David L. Donoho},
  title = {{Degrees of Freedom Analysis of Unrolled Neural Networks}},
  year = {2019},
  eprint = {1906.03742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.03742},
}
Figure