Abstract
Deep Unfolding Networks (DUNs) have emerged as a powerful paradigm for solving inverse problems by mapping iterative optimization algorithms into neural network layers. This paper establishes a formal link between DUNs and Ordinary Differential Equations (ODEs), revealing that standard unfolding layers are mathematically equivalent to first-order Forward Euler approximations. Leveraging this insight, we propose the High-Order Numerical Architecture (HNO) framework, which utilizes Linear Multi-step methods to design more efficient architectures. Our proposed 2NO and 3NO models demonstrate significantly faster convergence and superior accuracy across sparse coding, image inpainting, and compressive sensing benchmarks.
Introduction
The success of Deep Unfolding Networks (DUNs) lies in their ability to combine the structural interpretability of classical optimization with the high-capacity learning of deep neural networks. While effective, most existing DUNs are limited by their reliance on simple first-order iterative schemes. This research addresses a fundamental question: Can the rich mathematical history of numerical ODE solvers be used to systematically improve DUN design? By reinterpreting the iterative process as a continuous-time ODE, we can apply sophisticated numerical integration techniques to create more effective unfolding architectures.
Methodology: From ODEs to HNO
The core contribution of this work is the reinterpretation of a standard DUN layer, $x_{t+1} = \mathcal{G}(x_t | \Theta_t)$, as a discrete step in solving a first-order ODE: $\frac{dx(t)}{dt} = f(t, x | \Theta)$. In this view, the standard DUN is merely a Forward Euler approximation—the simplest and often least accurate numerical method.
To improve upon this, we introduce the High-Order Numerical Architecture (HNO) framework based on Linear Multi-step (LM) methods. Unlike first-order methods that only use the current state $x_t$, LM methods utilize information from multiple previous steps ($x_t, x_{t-1}, \dots, x_{t-r+1}$). We specifically develop:
- 2NO Architecture: A second-order scheme that incorporates $x_{t-1}$ and a learnable step size $\alpha_t$, providing a "momentum-like" update rule derived from ODE theory.
- 3NO Architecture: A third-order scheme that further utilizes $x_{t-2}$ for even higher precision.
These architectures are applied to state-of-the-art DUNs, including LISTA-CS for compressive sensing and GLISTA for gated sparse coding.
Experimental Evaluation
We evaluated the HNO framework on three primary tasks:
- Synthetic Sparse Coding: Testing theoretical convergence under varying noise levels (SNR) and condition numbers ($\kappa$).
- Natural Image Inpainting: Using the BSD500 dataset for training and Set11 for testing, simulating 50% pixel loss.
- Natural Image Compressive Sensing: Evaluating performance across compression ratios from 1% to 50% using the same datasets.
Key Results and Convergence Analysis
The theoretical advantage of higher-order methods translates directly into empirical performance. We prove that 2NO-LISTA-CS achieves a linear convergence rate that represents an "almost square improvement" over the original LISTA.
As shown in the convergence plots, the 2NO and 3NO models reach significantly lower Normalized Mean Squared Error (NMSE) values in fewer layers than their 1st-order counterparts. Key highlights include:
- Image Inpainting: On the Set11 benchmark, 2NO-GLISTA achieved a PSNR of 36.65 dB on the "House" image, outperforming standard GLISTA (35.93 dB) and LISTA (35.55 dB).
- Compressive Sensing: At a 10% CS ratio, 2NO-LISTA-CS achieved an average PSNR of 25.60 dB, surpassing the original LISTA-CS (25.20 dB) and traditional methods like D-AMP (22.64 dB).
Discussion and Future Directions
Our findings suggest that while moving from 1st-order to 2nd-order (2NO) provides a substantial performance leap, the transition to 3rd-order (3NO) yields diminishing returns relative to the increased computational cost. Thus, 2NO represents an optimal balance for practical applications.
Future research will focus on extending the HNO framework to multi-variable DUNs with internal memory states and exploring the potential of implicit numerical methods, which offer greater stability at the cost of increased per-layer complexity.
Summary updated