In data-driven science, discovering the physical laws governing a system requires calculating derivatives from experimental measurements. If you have a time-series of a planet's position or a grid of fluid velocities, finding the underlying differential equations requires knowing how those variables change over time and space.
However, numerical differentiation has a fundamental flaw: it is inherently unstable when paired with real-world, noisy data.
To understand why, consider a true signal $s(t)$ corrupted by a tiny amount of high-frequency noise:
$$f(t) = s(t) + \epsilon \sin(\omega t)$$
Even if the noise amplitude $\epsilon$ is microscopic, a high frequency $\omega$ means that when we take the derivative, the noise term scales up dramatically:
$$f'(t) = s'(t) + \epsilon \omega \cos(\omega t)$$
If we compute a second or third derivative, the noise is amplified exponentially ($\epsilon \omega^2$, $\epsilon \omega^3$), completely drowning out the true physical signal. This vulnerability makes direct numerical differentiation a major bottleneck for system identification and machine learning models that rely on physics-informed constraints.
To overcome this, a powerful family of techniques has emerged. Instead of directly differentiating noisy data, these methods either invert the problem into the integral domain or apply sophisticated pre-differentiation smoothing.
Strategy 1: Inverting the Problem via the Integral Domain
The most elegant way to avoid noise amplification is to avoid differentiation altogether. While differentiation multiplies the noise by its frequency ($\omega$), integration does the exact opposite: it divides by the frequency.
$$\int \epsilon \sin(\omega t) \, dt = -\frac{\epsilon}{\omega} \cos(\omega t)$$
As the frequency $\omega$ grows larger, the noise term shrinks toward zero. Integration acts as a natural, low-pass filter that suppresses high-frequency fluctuations.
Researchers exploit this property by converting differential equations into their weak form or integral representation before performing system identification. For example, in sparse regression frameworks like SINDy (Sparse Identification of Nonlinear Dynamics), Kaptanoglu et al. implement this in PySINDy: A comprehensive Python package for robust sparse system identification. By reformulating the governing equations into an integral representation, the sparse regression algorithm operates on integrated data, making the model discovery process highly resilient to sensor noise.
Similarly, Thanasutives et al. apply this concept to variable-coefficient partial differential equations (PDEs) in Bayesian model selection for variable-coefficient partial differential equation discovery. They integrate candidate terms over local spatio-temporal subdomains, transforming the differential terms into stable integral forms that prevent noise from corrupting the downstream best-subset regression.
This philosophy extends to fractional calculus as well. In Complete Identification of a Dynamic Fractional Order System Under Non-ideal Conditions Using Fractional Differintegral Definitions, Acharya et al. demonstrate that identifying fractional-order dynamical systems—which use non-integer derivatives—becomes significantly more robust when using fractional integration definitions rather than differentiation, naturally smoothing out high-frequency noise during parameter estimation.
Strategy 2: Pre-Differentiation Smoothing and Mollification
If an integral formulation is not feasible and derivatives must be estimated, the data must be cleaned before any finite-difference calculations occur. The goal is to construct a smooth, continuous representation of the data whose analytical derivatives can be safely computed.
Local Windowing and Regression
The simplest realization of this idea is to average data locally before sampling. Anderssen et al. proved the utility of this in A stable finite difference ansatz for higher order differentiation of non-exact data, showing that an "averaging-then-sampling" strategy over a local window acts as a regularization technique that stabilizes higher-order finite differences. Taking this further, Langley et al. developed the Regression-Guided Process Modeler (RPM) in Heuristic Induction of Rate-Based Process Models, which applies local regression smoothing to noisy time-series data before calculating derivatives, protecting the structural induction of the underlying model from noise-induced variance.
Splines and Mollifiers
For complex spatial data, splines provide a flexible way to enforce smoothness. In Identification of Underlying Partial Differential Equations from Noisy Data with Splines, Huo pairs computationally efficient cubic splines with Lasso regression to accurately recover governing PDEs without amplifying observational noise.
In deep learning contexts, automatic differentiation (autodiff) can be notoriously sensitive to noise. To fix this, Vinayak et al. introduced "Mollifier Layers" in Mollifier Layers: Enabling Efficient High-Order Derivatives in Inverse PDE Learning. These layers replace standard autodiff with convolutional operations using smooth mollifier kernels. Because convolution with a smooth kernel is mathematically equivalent to an integration process, it filters out high-frequency noise, making high-order derivative estimation stable and fast.
Noisy Data ──> [ Mollifier / Smoothing Kernel ] ──> Smooth Representation ──> Safe Differentiation
Neural Network Pre-smoothing
Neural networks themselves can serve as powerful smoothers. In DISCOVER: Deep identification of symbolically concise open-form partial differential equations via enhanced reinforcement learning, Du et al. use a neural network as a preprocessing step to fit raw experimental data. The network acts as a low-pass filter, outputting a continuous, clean representation from which robust numerical derivatives can be extracted for symbolic PDE discovery.
Strategy 3: Successive and Iterative Denoising
When dealing with exceptionally high levels of noise or higher-order derivatives, a single preprocessing step may not be enough. Noise can creep back into the system during intermediate calculations.
To combat this, He et al. proposed a technique called Successive Denoised Differentiation (SDD) in Robust Identification of Differential Equations by Numerical Techniques from a Single Set of Noisy Observation. Instead of smoothing just once, the authors apply a smoothing operator (such as Moving Least Squares) iteratively before and after each individual differentiation step. This multi-stage pipeline prevents the compounding amplification of noise across successive derivatives.
The authors adapt this multi-step smoothing philosophy further in Group projected Subspace Pursuit for Identification of variable coefficient differential equations (GP-IDENT), where He 0001 et al. combine B-spline approximations with successive denoising to stabilize the identification of PDEs whose coefficients vary across space and time.
The Universality of Smoothness Constraints
The core mathematical insight behind these methods—that enforcing smoothness or filtering data mitigates noise amplification—extends far beyond physical system identification:
- Speech Processing: In Least squares signal declipping for robust speech recognition, Harvilla et al. show that incorporating derivative and smoothness constraints into a least-squares minimization framework allows for the robust reconstruction of clipped audio signals, even in heavily corrupted environments.
- Machine Learning Labels: In IRNet: Iterative Refinement Network for Noisy Partial Label Learning, Lian et al. use explicit smoothness constraints to ensure that minor variations in input data do not cause erratic jumps in model predictions, stabilizing classification when training labels are noisy or incorrect.
- Spherical Data Approximation: In Distributed Filtered Hyperinterpolation for Noisy Data on the Sphere, Wang 0001 et al. use a filtered hyperinterpolation strategy to isolate and approximate underlying smooth functions from noisy spherical data distributed across parallel servers.
Summary
Numerical differentiation inherently magnifies errors because it scales with the frequency of the noise. By shifting the mathematical framing to the integral domain (where noise is divided by its frequency) or by systematically applying smoothing operations (splines, moving least squares, or mollifiers) before computing changes, these modern frameworks turn an ill-posed, unstable problem into a robust, reliable tool for scientific discovery.
Key Foundational Work
Complete Identification of a Dynamic Fractional Order System Under Non-ideal Conditions Using Fractional Differintegral Definitions (2008) — contributed 1 concept elements to this direction.
A stable finite difference ansatz for higher order differentiation of non-exact data (1998) — contributed 1 concept elements to this direction.