Scientific modeling has historically been divided into two camps. On one side are mechanistic models—ordinary and partial differential equations (ODEs/PDEs) derived from first principles like the conservation of energy or mass. These models are highly interpretable and generalize well, but they are often "incomplete," failing to capture complex, unmodeled effects like friction, biological nuances, or sub-grid turbulence.
On the other side is machine learning, specifically deep neural networks. These are "universal approximators" capable of learning any function from data, but they are notoriously data-hungry, lack physical constraints, and often behave as "black boxes" that fail when extrapolated beyond their training set.
The emerging field of Scientific Machine Learning (SciML) bridges this gap by embedding neural networks directly into the functional form of differential equations. This "grey-box" approach allows researchers to keep the physics they know while using data to "fill in the blanks" for the parts they don't.
The Mechanism: Universal Differential Equations
The core innovation, popularized by Christopher Vincent Rackauckas, Alan Edelman, and their team in Universal Differential Equations for Scientific Machine Learning, is the Universal Differential Equation (UDE).
In a traditional mechanistic model, we might write: $$\frac{dx}{dt} = f(x, t, \theta)$$ where $f$ is a known mathematical function. In a UDE, we augment this structure: $$\frac{dx}{dt} = f_{known}(x, t, \theta) + NN(x, t, \phi)$$ Here, $NN$ is a neural network with weights $\phi$. The neural network doesn't replace the model; it acts as a specific term within the derivative, representing an unknown force, a missing chemical reaction, or a complex social behavior.
This architecture provides a structural prior. Because the neural network is constrained within a differential equation, the resulting model must still satisfy the fundamental "logic" of the system. For example, in Learning Absorption Rates in Glucose-Insulin Dynamics from Meal Covariates, Ke Alexander Wang and Emily Fox use neural networks to learn personalized absorption profiles while maintaining the underlying physiological ODEs.
Why It Works: Differentiable Programming
The reason we can train these hybrid models is differentiable programming. To optimize the neural network weights, we need to calculate how the model's error changes with respect to those weights. This requires "backpropagating" through the entire differential equation solver.
Christopher Vincent Rackauckas and Mike Innes developed DiffEqFlux.jl to treat differential equation solvers as differentiable layers. By making the solver itself part of the neural network's computational graph, the system can learn the unknown parameters of a dynamical system even from sparse or noisy data.
This approach is significantly more data-efficient than pure deep learning. Because the "known physics" handles the heavy lifting of the system's structure, the neural network only needs to learn the residual dynamics. As demonstrated by Wannes De Groote in Neural Network Augmented Physics Models for Systems With Partially Unknown Dynamics, this allows for the discovery of complex nonlinearities, like friction in mechanical linkages, using only standard sensor data.
From Prediction to Discovery
One of the most powerful facets of this research direction is its ability to move beyond mere prediction toward scientific discovery. Once a UDE is trained, the learned neural network term can often be "distilled" back into a symbolic mathematical expression.
In fluid dynamics, Balu Nadiga and colleagues showed in Interpreting neural network models of residual scalar flux that neural networks trained on turbulence data don't just fit the noise; they autonomously discover hybrid modeling strategies that combine structural reconstruction with functional diffusion.
This "grey-box" philosophy is now being applied across diverse domains:
- Systems Biology: Paul F. Lang and Christopher Vincent Rackauckas use SBMLToolkit.jl to integrate traditional biological pathways with neural ODEs to fill gaps in kinetic laws.
- Epidemiology: Researchers are augmenting SIR models with neural networks to account for time-varying transmission rates driven by weather or policy changes, as seen in Neural Network Augmented Compartmental Pandemic Models.
- Hydrology: Marvin Höge and Fabrizio Fenicia demonstrate in Improving hydrologic models for predictions and process understanding using neural ODEs that replacing rigid conceptual equations with flexible neural components maintains interpretability while matching the accuracy of black-box models.
The Future: Digital Twins and Causal Validity
As the field matures, the focus is shifting toward ensuring these models remain causally consistent. Matthew E. Levine and Emily Fox introduced the $H^2NCM$ framework in Hybrid$^2$ Neural ODE Causal Modeling, which uses a "causal loss" to ensure that hybrid models respect the known ranking of treatment effects.
By combining the rigor of differential equations with the flexibility of neural networks, SciML is moving toward "Digital Twins" that can not only predict the future but also explain why it happens and how to intervene. Whether reconstructing tumor growth from PSA tests (Daniel Camacho-Gómez) or accelerating climate simulations by 10,000x (Sharv Murgai), the integration of neural networks into the governing equations of science is transforming how we model the natural world.