HNO: High-Order Numerical Architecture for ODE-Inspired Deep Unfolding Networks

Lin Kong, Wei Sun, Fanhua Shang, Yuanyuan Liu, Hongying Liu

Abstract

Deep Unfolding Networks (DUNs) have emerged as a powerful paradigm for solving inverse problems by mapping iterative optimization algorithms into neural network layers. This paper establishes a formal link between DUNs and Ordinary Differential Equations (ODEs), revealing that standard unfolding layers are mathematically equivalent to first-order Forward Euler approximations. Leveraging this insight, we propose the High-Order Numerical Architecture (HNO) framework, which utilizes Linear Multi-step methods to design more efficient architectures. Our proposed 2NO and 3NO models demonstrate significantly faster convergence and superior accuracy across sparse coding, image inpainting, and compressive sensing benchmarks.

Introduction

The success of Deep Unfolding Networks (DUNs) lies in their ability to combine the structural interpretability of classical optimization with the high-capacity learning of deep neural networks. While effective, most existing DUNs are limited by their reliance on simple first-order iterative schemes. This research addresses a fundamental question: Can the rich mathematical history of numerical ODE solvers be used to systematically improve DUN design? By reinterpreting the iterative process as a continuous-time ODE, we can apply sophisticated numerical integration techniques to create more effective unfolding architectures.

Methodology: From ODEs to HNO

The core contribution of this work is the reinterpretation of a standard DUN layer, $x_{t+1} = \mathcal{G}(x_t | \Theta_t)$, as a discrete step in solving a first-order ODE: $\frac{dx(t)}{dt} = f(t, x | \Theta)$. In this view, the standard DUN is merely a Forward Euler approximation—the simplest and often least accurate numerical method.

Evolution of the HNO framework from standard Deep Unfolding Networks via ODE reinterpretation.

To improve upon this, we introduce the High-Order Numerical Architecture (HNO) framework based on Linear Multi-step (LM) methods. Unlike first-order methods that only use the current state $x_t$, LM methods utilize information from multiple previous steps ($x_t, x_{t-1}, \dots, x_{t-r+1}$). We specifically develop:

  • 2NO Architecture: A second-order scheme that incorporates $x_{t-1}$ and a learnable step size $\alpha_t$, providing a "momentum-like" update rule derived from ODE theory.
  • 3NO Architecture: A third-order scheme that further utilizes $x_{t-2}$ for even higher precision.

These architectures are applied to state-of-the-art DUNs, including LISTA-CS for compressive sensing and GLISTA for gated sparse coding.

Experimental Evaluation

We evaluated the HNO framework on three primary tasks:

  1. Synthetic Sparse Coding: Testing theoretical convergence under varying noise levels (SNR) and condition numbers ($\kappa$).
  2. Natural Image Inpainting: Using the BSD500 dataset for training and Set11 for testing, simulating 50% pixel loss.
  3. Natural Image Compressive Sensing: Evaluating performance across compression ratios from 1% to 50% using the same datasets.

Key Results and Convergence Analysis

The theoretical advantage of higher-order methods translates directly into empirical performance. We prove that 2NO-LISTA-CS achieves a linear convergence rate that represents an "almost square improvement" over the original LISTA.

Convergence performance (NMSE vs. Number of Layers) comparing standard DUNs with their 2NO and 3NO counterparts.

As shown in the convergence plots, the 2NO and 3NO models reach significantly lower Normalized Mean Squared Error (NMSE) values in fewer layers than their 1st-order counterparts. Key highlights include:

  • Image Inpainting: On the Set11 benchmark, 2NO-GLISTA achieved a PSNR of 36.65 dB on the "House" image, outperforming standard GLISTA (35.93 dB) and LISTA (35.55 dB).
  • Compressive Sensing: At a 10% CS ratio, 2NO-LISTA-CS achieved an average PSNR of 25.60 dB, surpassing the original LISTA-CS (25.20 dB) and traditional methods like D-AMP (22.64 dB).

Discussion and Future Directions

Our findings suggest that while moving from 1st-order to 2nd-order (2NO) provides a substantial performance leap, the transition to 3rd-order (3NO) yields diminishing returns relative to the increased computational cost. Thus, 2NO represents an optimal balance for practical applications.

Future research will focus on extending the HNO framework to multi-variable DUNs with internal memory states and exploring the potential of implicit numerical methods, which offer greater stability at the cost of increased per-layer complexity.

Summary updated

Abstract

Recently, deep unfolding networks (DUNs) based on optimization algorithms have received increasing attention, and their high efficiency has been confirmed by many experimental and theoretical results. Since this type of networks combines model-based traditional optimization algorithms, they have high interpretability. In addition, ordinary differential equations (ODEs) are often used to explain deep neural networks, and provide some inspiration for designing innovative network models. In this paper, we transform DUNs into first-order ODE forms, and propose a high-order numerical architecture for ODE-inspired deep unfolding networks. To the best of our knowledge, this is the first work to establish the relationship between DUNs and ODEs. Moreover, we take two representative DUNs as examples, apply our architecture to them and design novel DUNs. In theory, we prove the existence, uniqueness of the solution and convergence of the proposed network, and also prove that our network obtains a fast linear convergence rate. Extensive experiments verify the effectiveness and advantages of our architecture.
BibTeX
@inproceedings{kong2022hno,
  author = {Lin Kong and Wei Sun and Fanhua Shang and Yuanyuan Liu and Hongying Liu},
  title = {{HNO: High-Order Numerical Architecture for ODE-Inspired Deep Unfolding Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i7.20683},
}
Figure