Noise Amplification in Numerical Differentiation

505 concepts381 papers2 proposals

When discovering the physical laws governing a dynamical system from data, we often need to know how fast a variable changes over time or space. Mathematically, this means computing derivatives. However, when working with real-world experimental data, computing these derivatives numerically acts as a massive noise amplifier, frequently destroying the underlying physical signal.

The Intuition: Why Differentiation Blows Up Noise

To understand why numerical differentiation is inherently unstable, consider a true physical signal $f(t)$ that is corrupted by a tiny amount of high-frequency measurement noise $\eta(t)$. We observe the combined signal:

$$y(t) = f(t) + \eta(t)$$

Suppose the noise is a simple high-frequency ripple with a very small amplitude $\epsilon$ and a high frequency $\omega$:

$$\eta(t) = \epsilon \sin(\omega t)$$

Because $\epsilon$ is tiny, the observed signal $y(t)$ looks almost identical to the clean signal $f(t)$. But watch what happens when we take the derivative with respect to time:

$$y'(t) = f'(t) + \epsilon \omega \cos(\omega t)$$

Even if the noise amplitude is minuscule (e.g., $\epsilon = 0.001$), a high frequency (e.g., $\omega = 10,000$, typical for electronic sensor jitter) causes the noise term to be multiplied by $\omega$. The noise contribution in the derivative becomes $0.001 \times 10,000 = 10$, completely overwhelming the true physical derivative $f'(t)$.

As shown by Bob Anderssen et al. in A stable finite difference ansatz for higher order differentiation of non-exact data, this amplification worsens exponentially with the order of the derivative. Computing second-, third-, or fourth-order derivatives from discrete, non-exact data quickly becomes an ill-posed mathematical problem.

How Noise Amplification Breaks Model Discovery

Many modern scientific machine learning frameworks rely on point-by-point numerical differentiation to extract equations from data. For instance, sparse regression methods like SINDy (Brian M. de Silva et al. in PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data) and parametric PDE discovery tools (Samuel Rudy et al. in Data-driven identification of parametric partial differential equations) construct a dictionary of candidate terms (such as $u, u^2, \nabla^2 u$) and try to regress them against estimated time derivatives.

When data contains noise, this dependency causes model discovery to fail in several predictable ways:

Alternative Paradigms: Bypassing Finite Differences

To build robust discovery algorithms, researchers have developed several core strategies to avoid direct, point-by-point numerical differentiation.

1. The Weak Form and Integration by Parts

Instead of forcing a model to match a derivative at every discrete point, one can integrate the governing equations over a local region. By applying integration by parts, the derivative operator is transferred away from the noisy data and onto a smooth, analytically defined "test function."

For example, Kevin Course et al. in Weak Form Generalized Hamiltonian Learning utilize a weak form loss function to learn Generalized Hamiltonian Neural Networks, achieving a 45x speedup over adjoint methods and exceptional noise robustness. Similarly, the D-CIPHER framework developed by Krzysztof Kacprzyk et al. in D-CIPHER: Discovery of Closed-form Partial Differential Equations leverages a variational formulation to discover closed-form PDEs directly from noisy observations without explicit derivative estimation. For fractional-order systems, Ayan Acharya demonstrates in Complete Identification of a Dynamic Fractional Order System Under Non-ideal Conditions Using Fractional Differintegral Definitions that substituting fractional differentiation with fractional integration provides a highly stable parameter identification framework under non-ideal conditions.

2. Differentiable Denoising and Splines

Another strategy fits a smooth, continuous mathematical representation to the data first, then differentiates that representation analytically rather than differentiating raw data points.

Fangzheng Sun et al. in Physics-informed Spline Learning for Nonlinear Dynamics Discovery introduce Physics-informed Spline Learning (PiSL), which combines the local flexibility of B-splines with global sparse regression constraints to calculate reliable velocities from noisy or incomplete data. This approach is extended by Luning Sun et al. in Bayesian Spline Learning for Equation Discovery of Nonlinear Dynamics with Quantified Uncertainty into a Bayesian spline framework, enabling equation discovery with rigorous uncertainty quantification.

3. Continuous-Time Trajectory Matching

Instead of matching derivatives (gradient matching), models can be evaluated by integrating the predicted dynamics forward in time and comparing the resulting trajectory directly to the observed data.

Alexander Ilin et al. in Learning Trajectories of Hamiltonian Systems with Neural Networks treat the system trajectory as a continuous, learnable function of time, allowing models to converge accurately even when data is noisy or irregularly sampled. In a similar vein, Mohamed Aziz Bhouri et al. in Gaussian processes meet NeuralODEs: A Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data combine Gaussian Processes with Neural ODEs to bypass raw numerical derivatives entirely when data is scarce and corrupted.

4. Learning Discrete Maps and Invariant Features

Finally, some methods avoid continuous-time equations altogether, focusing instead on discrete-time mappings or global physical properties.

Molei Tao et al. in Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps develop Generating Function Neural Networks (GFNNs) that learn exactly-symplectic discrete maps, eliminating the need to estimate instantaneous rates of change. Alternatively, Kazem Meidani et al. in Data-driven Identification of 2D Partial Differential Equations using extracted physical features identify governing PDEs by extracting global physical features like symmetry and frequency rather than computing local numerical derivatives, allowing the model to accurately generalize to entirely unseen physics.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 359 papers
  2. Version 1 · 2026-10-04 · 442 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_numerical_differentiation_instability_63cde1dd,
  title = {numerical differentiation instability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-differentiation-instability/rdr_63cde1dd0d054a629878bf43d6cfb6f9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_numerical_differentiation_instability_63cde1dd_v2,
  title = {numerical differentiation instability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-differentiation-instability/rdr_63cde1dd0d054a629878bf43d6cfb6f9/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake