Learning Residual Dynamics via Hybrid Differential Equations

1,038 concepts638 papers8 proposals

When modeling complex real-world systems—whether tracking the spread of a pandemic, simulating fluid flow, or predicting the movement of a mechanical actuator—we often face a frustrating dilemma.

On one hand, we have first-principles physics models (such as ordinary or partial differential equations). These models are highly interpretable, data-efficient, and strictly respect fundamental conservation laws. However, they are almost always incomplete. They rely on simplifying assumptions that omit messy, real-world nuances like unmodeled friction, sub-grid turbulence, or shifting human behaviors.

On the other hand, we have purely data-driven neural networks. These black-box models possess the high capacity needed to fit arbitrarily complex, non-linear patterns from data. Yet, they are notoriously data-hungry, generalize poorly outside their training distribution, and frequently violate basic physical realities (such as creating mass or energy out of thin air).

The hybrid modeling paradigm—often realized through frameworks like Universal Differential Equations (UDEs) or Neural Network Augmented Physics (NNAP)—bridges this gap. Instead of forcing a neural network to learn an entire physical system from scratch, this approach embeds neural networks directly inside the governing differential equations. By letting physics handle what we already know, the neural network is freed to focus exclusively on learning the unknown residuals.


The Additive Residual Architecture

The most intuitive way to construct a hybrid model is through an additive decomposition of the system's derivative. Mathematically, instead of modeling a dynamical system as a pure black box, we split its temporal evolution into two distinct forces:

$$\frac{dx}{dt} = f_{\text{physics}}(x) + f_{\text{neural}}(x; \theta)$$

Here, $f_{\text{physics}}(x)$ represents the known, simplified mathematical model of the system, while $f_{\text{neural}}(x; \theta)$ is a neural network parameterized by weights $\theta$.

This structure works because it drastically reduces the search space for the neural network. Rather than learning the baseline trajectories or basic conservation laws, the network only needs to learn the discrepancy between theory and reality.

For instance, Victoriya Kashtanova et al. utilize this additive correction framework in Simultaneous data assimilation and cardiac electrophysiology model correction using differentiable physics and deep learning to correct simplified biological equations governing the heart's electrical activity. By focusing the neural network solely on the residual dynamics, the authors achieve significantly higher data efficiency and better generalization than purely data-driven alternatives.

Similarly, Manuel Haussmann applies this additive combination within a stochastic framework in Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes. The author demonstrates that even when the underlying physical ordinary differential equations are vague or incorrectly parameterized, they still provide a powerful inductive bias that prevents the neural network from overfitting.

In mechanical engineering, this architecture prevents models from making physically impossible predictions when operating outside nominal conditions. Wannes De Groote et al. demonstrate this in Neural Network Augmented Physics Models for Systems With Partially Unknown Dynamics: Application to Slider–Crank Mechanism by isolating unmodeled friction and complex loads into a neural module. Because the core physics layer remains explicit, the model retains its structural interpretability while capturing complex, non-linear real-world behaviors.


Functional Parameterization and Closure Terms

In many complex systems, the missing piece of the puzzle is not an additive external force, but rather a hidden, time-varying parameter or an unresolved sub-grid interaction. In these scenarios, the neural network is embedded inside the physical equations to act as a functional parameterizer or a "closure term":

$$\frac{dx}{dt} = f_{\text{physics}}\big(x, \mu(t, x)\big) \quad \text{where} \quad \mu(t, x) = f_{\text{neural}}(t, x; \theta)$$

This formulation is particularly dominant in fields like epidemiology and climate science, where the overarching structural transitions are well-understood, but the driving parameters are highly volatile and dependent on external data.

In epidemiological modeling, Mohamed Aziz Bhouri et al. apply this strategy in COVID-19 dynamics across the US: A deep learning study of human mobility and social behavior. Instead of using a neural network to predict infection counts directly, the authors use an ensemble of networks to map human mobility and behavioral data to time-varying disease transmission rates. These rates are then fed directly into a classical compartmental SEIR differential equation system. This allows the model to capture complex, data-driven behavioral influences while strictly maintaining the structural interpretability and physical constraints of traditional epidemiology.

Emma Wang adopts a parallel approach in Implications of delayed reopening in controlling the COVID-19 surge in Southern and West-Central USA, augmenting compartmental differential equations with neural network modules to dynamically learn shifting quarantine strengths directly from observed data without manual parameter tuning.

This philosophy extends naturally to systems biology. As Maren Philipps et al. outline in Current state and open problems in universal differential equations for systems biology, embedding neural networks within mechanistic models allows researchers to approximate unknown biological interactions or chemical kinetics. The resulting "grey-box" models leverage known biological pathways while using data-driven components to fill critical gaps in scientific knowledge.


End-to-End Training via Differentiable Physics

The primary technical challenge of hybrid modeling is optimization: How do we train a neural network whose outputs are buried inside a differential equation solver?

We rarely have direct supervision for the network's immediate outputs (such as the instantaneous residual force or the exact parameter value). Instead, we only observe the system's state $x(t)$ at discrete time intervals. If we try to train the network by evaluating its errors step-by-step in isolation, small errors quickly compound over time, causing the simulated trajectory to drift catastrophically away from reality.

To solve this, hybrid modeling relies on differentiable programming. By treating the entire numerical differential equation solver as a sequence of differentiable operations, gradients can flow backward from the final trajectory loss function, through the integration steps of the solver, and directly into the neural network weights $\theta$.

[ Neural Network: f_neural(x; θ) ] 
               │
               ▼ (Predicted Residual / Parameter)
[ Physics Core: f_physics ] ──► [ Differentiable Solver ] ──► [ Predicted Trajectory x(t) ]
                                                                        │
                                                                        ▼ (Loss Evaluation)
 Gradient Flow ◄───────────────────────────────────────────────────────┘

This end-to-end optimization ensures that the neural network learns to cooperate with the physics engine over the entire time horizon. [Yibo Yang et al. leverage this paradigm in Bayesian differential programming for robust systems identification under uncertainty. By combining a structured dictionary of known physical functions with a neural network closure term, the authors use differentiable programming to backpropagate trajectory errors, allowing for the robust identification of underlying dynamics even when the available data is noisy, sparse, or irregularly sampled.

Furthermore, integrating neural networks directly into the solver loop allows for the creation of smart numerical correctors. For example, Zhongzhan Huang et al. introduce a hybrid framework in On fast simulation of dynamical system with neural vector enhanced numerical solver where a neural network acts as a learned corrector vector. Trained end-to-end, the network learns to predict and compensate for the integration errors that occur when a traditional solver takes large, coarse time steps. This allows simulations to run orders of magnitude faster while maintaining the structural stability and accuracy of fine-grained numerical solvers.

Why It Works: Bounded Extrapolation

The ultimate power of hybrid physics-neural modeling lies in its safety and extrapolation capabilities. When a pure neural network encounters data outside its training distribution, its predictions become highly unpredictable.

In a hybrid model, if the neural network encounters an unfamiliar state, its residual output may degrade, but the overall system remains anchored by the explicit physical equations. The core physics acts as a structural prior, bounding the system's behavior and ensuring that even when the network's predictions are imperfect, the model as a whole still behaves like a physical system.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_residual_dynamics_learning_60dcca1a,
  title = {residual dynamics learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/residual-dynamics-learning/rdr_60dcca1a84654d8892e74fcdc3b4046a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_residual_dynamics_learning_60dcca1a_v1,
  title = {residual dynamics learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/residual-dynamics-learning/rdr_60dcca1a84654d8892e74fcdc3b4046a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake