Abstract
Forecasting physical signals in long time range is among the most challenging tasks in Partial Differential Equations (PDEs) research. To circumvent limitations of traditional solvers, many different Deep Learning methods have been proposed. They are all based on auto-regressive methods and exhibit stability issues. Drawing inspiration from the stability property of implicit numerical schemes, we introduce a stable auto-regressive implicit neural network. We develop a theory based on the stability definition of schemes to ensure the stability in forecasting of this network. It leads us to introduce hard constraints on its weights and propagate the dynamics in the latent space. Our experimental results validate our stability property, and show improved results at long-term forecasting for two transports PDEs.
The Challenge of Long-Term Forecasting
Predicting the behavior of dynamical systems described by Partial Differential Equations (PDEs)—such as fluid flow or climate patterns—is a cornerstone of scientific computing. While Deep Learning (DL) models like Fourier Neural Operators (FNOs) and ResNets have shown great promise, they are almost always "auto-regressive." This means the model’s output at one time step becomes its input for the next.
The fatal flaw of this approach is instability. Small errors in the first few steps accumulate and amplify, eventually leading the model to produce "garbage" results or infinite values (divergence). While traditional numerical solvers use "implicit schemes" to maintain stability, most DL architectures are "explicit," making them prone to these blow-ups. This research bridges the gap by bringing the theoretical stability of implicit numerical analysis into the world of neural networks.
Implicit ResNets: Stability by Design
The proposed Implicit ResNet architecture differs from standard ResNets where the next state is a direct function of the current state ($x_{n+1} = x_n + f(x_n)$). Instead, an implicit network defines the next state through an equation that must be solved: $x_{n+1} = x_n + R_n(x_{n+1})$.
Key technical innovations include:
- Hard Weight Constraints: To guarantee stability, the authors prove that if the weight matrix $W_n$ is upper triangular and its diagonal elements stay within the range $[-1, 0)$, the network is stable in the $L^p$ norm. These constraints are enforced during training.
- Latent Space Forecasting: To maintain efficiency, the model uses an encoder to project high-dimensional physical data into a smaller "latent space," performs the stable forecasting there, and then decodes it back.
- Differentiable Root-Finding: Because the implicit step requires solving an equation, the system uses Broyden’s method (a quasi-Newton solver) within the network, allowing the entire system to remain end-to-end trainable.
Experimental Validation
The model was evaluated on two classic 1D transport problems: the linear Advection Equation and the non-linear Burgers’ Equation. Models were trained only on a single time step transition but were required to forecast up to 400 steps into the future.
As seen in the figure above, explicit models (Explicit ResNet and FNO) exhibit a convex error curve, where failure accelerates over time. In contrast, the Implicit ResNet reaches a "stable plateau," preventing the simulation from ever blowing up.
Key Results
The results highlight a stark difference in long-term reliability. While standard models achieve lower training and test errors (short-term), they fail catastrophically as the forecast horizon extends.
- Advection: At the 400-step mark, the Explicit ResNet reached an astronomical Mean Squared Error (MSE) of $10^{31}$. The proposed Implicit ResNet stayed stable at an MSE of 27.5.
- Burgers’: While the baselines diverged to infinity ($+\infty$), the Implicit ResNet maintained a stable MSE of 0.66.
The Stability-Accuracy Trade-off
A major insight from this work is the trade-off between short-term precision and long-term stability. While baselines like FNO are more accurate initially, they lack a "safety floor." The implicit model is less precise at the start but provides structural guarantees that make it the only viable option for very long horizons.
Future work will focus on improving initial accuracy without sacrificing stability, reducing the computational cost of the iterative root-finding solver, and expanding the theory to more complex, high-dimensional 3D physics.
Summary updated