When discovering the physical laws governing a dynamical system from data, we often need to know how fast a variable changes over time or space. Mathematically, this means computing derivatives. However, when working with real-world experimental data, computing these derivatives numerically acts as a massive noise amplifier, frequently destroying the underlying physical signal.
The Intuition: Why Differentiation Blows Up Noise
To understand why numerical differentiation is inherently unstable, consider a true physical signal $f(t)$ that is corrupted by a tiny amount of high-frequency measurement noise $\eta(t)$. We observe the combined signal:
$$y(t) = f(t) + \eta(t)$$
Suppose the noise is a simple high-frequency ripple with a very small amplitude $\epsilon$ and a high frequency $\omega$:
$$\eta(t) = \epsilon \sin(\omega t)$$
Because $\epsilon$ is tiny, the observed signal $y(t)$ looks almost identical to the clean signal $f(t)$. But watch what happens when we take the derivative with respect to time:
$$y'(t) = f'(t) + \epsilon \omega \cos(\omega t)$$
Even if the noise amplitude is minuscule (e.g., $\epsilon = 0.001$), a high frequency (e.g., $\omega = 10,000$, typical for electronic sensor jitter) causes the noise term to be multiplied by $\omega$. The noise contribution in the derivative becomes $0.001 \times 10,000 = 10$, completely overwhelming the true physical derivative $f'(t)$.
As shown by Bob Anderssen et al. in A stable finite difference ansatz for higher order differentiation of non-exact data, this amplification worsens exponentially with the order of the derivative. Computing second-, third-, or fourth-order derivatives from discrete, non-exact data quickly becomes an ill-posed mathematical problem.
How Noise Amplification Breaks Model Discovery
Many modern scientific machine learning frameworks rely on point-by-point numerical differentiation to extract equations from data. For instance, sparse regression methods like SINDy (Brian M. de Silva et al. in PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data) and parametric PDE discovery tools (Samuel Rudy et al. in Data-driven identification of parametric partial differential equations) construct a dictionary of candidate terms (such as $u, u^2, \nabla^2 u$) and try to regress them against estimated time derivatives.
When data contains noise, this dependency causes model discovery to fail in several predictable ways:
- Structural Misidentification: Because minor measurement errors lead to massive inaccuracies in estimated derivatives, regression algorithms select incorrect terms or construct physically impossible model structures, as detailed by Zhaozhi Qian et al. in D-CODE: Discovering Closed-form ODEs from Observed Trajectories and Pat Langley et al. in Heuristic Induction of Rate-Based Process Models.
- Vanishing High-Order Terms: In complex systems governed by high-order spatial derivatives, the signal-to-noise ratio degrades so rapidly that the true physical terms become entirely obscured, making it difficult to identify equations that rely on terms like fourth-order derivatives (Samuel Rudy et al. in Data-driven identification of parametric partial differential equations).
- Control and Reinforcement Learning Instabilities: In robotics, calculating joint accelerations from noisy kinematics introduces severe artifacts that degrade inverse dynamics models (Diego Romeres et al. in Online semi-parametric learning for inverse dynamics modeling). Similarly, using finite differences to approximate continuous-time dynamics from discrete transitions introduces erroneous gradient estimations that destabilize value function regularization (Yun-Hsuan Lien et al. in Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning).
Alternative Paradigms: Bypassing Finite Differences
To build robust discovery algorithms, researchers have developed several core strategies to avoid direct, point-by-point numerical differentiation.
1. The Weak Form and Integration by Parts
Instead of forcing a model to match a derivative at every discrete point, one can integrate the governing equations over a local region. By applying integration by parts, the derivative operator is transferred away from the noisy data and onto a smooth, analytically defined "test function."
For example, Kevin Course et al. in Weak Form Generalized Hamiltonian Learning utilize a weak form loss function to learn Generalized Hamiltonian Neural Networks, achieving a 45x speedup over adjoint methods and exceptional noise robustness. Similarly, the D-CIPHER framework developed by Krzysztof Kacprzyk et al. in D-CIPHER: Discovery of Closed-form Partial Differential Equations leverages a variational formulation to discover closed-form PDEs directly from noisy observations without explicit derivative estimation. For fractional-order systems, Ayan Acharya demonstrates in Complete Identification of a Dynamic Fractional Order System Under Non-ideal Conditions Using Fractional Differintegral Definitions that substituting fractional differentiation with fractional integration provides a highly stable parameter identification framework under non-ideal conditions.
2. Differentiable Denoising and Splines
Another strategy fits a smooth, continuous mathematical representation to the data first, then differentiates that representation analytically rather than differentiating raw data points.
Fangzheng Sun et al. in Physics-informed Spline Learning for Nonlinear Dynamics Discovery introduce Physics-informed Spline Learning (PiSL), which combines the local flexibility of B-splines with global sparse regression constraints to calculate reliable velocities from noisy or incomplete data. This approach is extended by Luning Sun et al. in Bayesian Spline Learning for Equation Discovery of Nonlinear Dynamics with Quantified Uncertainty into a Bayesian spline framework, enabling equation discovery with rigorous uncertainty quantification.
3. Continuous-Time Trajectory Matching
Instead of matching derivatives (gradient matching), models can be evaluated by integrating the predicted dynamics forward in time and comparing the resulting trajectory directly to the observed data.
Alexander Ilin et al. in Learning Trajectories of Hamiltonian Systems with Neural Networks treat the system trajectory as a continuous, learnable function of time, allowing models to converge accurately even when data is noisy or irregularly sampled. In a similar vein, Mohamed Aziz Bhouri et al. in Gaussian processes meet NeuralODEs: A Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data combine Gaussian Processes with Neural ODEs to bypass raw numerical derivatives entirely when data is scarce and corrupted.
4. Learning Discrete Maps and Invariant Features
Finally, some methods avoid continuous-time equations altogether, focusing instead on discrete-time mappings or global physical properties.
Molei Tao et al. in Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps develop Generating Function Neural Networks (GFNNs) that learn exactly-symplectic discrete maps, eliminating the need to estimate instantaneous rates of change. Alternatively, Kazem Meidani et al. in Data-driven Identification of 2D Partial Differential Equations using extracted physical features identify governing PDEs by extracting global physical features like symmetry and frequency rather than computing local numerical derivatives, allowing the model to accurately generalize to entirely unseen physics.