Many physical phenomena and geometric transformations are naturally expressed using continuous partial differential equations (PDEs). Whether modeling fluid dynamics, quantum mechanics, or the rotation of an image, partial differential operators (PDOs) provide a coordinate-independent language for describing local changes.
However, computers do not operate on continuous spaces; they process discrete grids—be they simulation meshes, 2D pixel arrays, or 3D voxel grids. To bridge this gap, continuous operators must be mapped onto discrete structures. The choice of how we perform this discretization determines whether the underlying mathematical symmetries of our world are preserved or lost.
The Limits of Finite Differences
The traditional workhorse for approximating derivatives on a grid is the Finite Difference Method (FDM). In FDM, derivatives are calculated by taking the differences between values at neighboring grid points. This approach is foundational across scientific computing, powering electronic structure simulations in packages like Octopus (Andrade et al.) and SPARC (Xu et al.), liquid crystal modeling (Mei et al.), and plasma transport simulations (Hornsby et al.).
In deep learning, early frameworks for geometric equivariance—ensuring a network responds predictably to transformations like rotations—also relied on finite differences. For example, Shen et al. introduced PDO-eConvs, which mapped continuous differential operators to pixel grids using finite differences to build rotation-equivariant convolutional networks.
Finite Difference (Grid-Aligned):
[ -1 ] --- [ 0 ] --- [ +1 ] --> Highly sensitive to grid orientation
While computationally simple, finite differences suffer from a fundamental flaw when applied to continuous symmetries: they are inherently tied to the grid's axes. If you rotate an image by exactly 90 degrees, a standard finite difference stencil aligns perfectly with the new pixel boundaries. But if you rotate it by an arbitrary angle, like 33 degrees, the target points no longer land on exact grid coordinates.
As the authors of later frameworks noted, finite difference approximations degrade severely under arbitrary rotations, restricting exact equivariance to rigid, grid-aligned transformations (Shen et al.). High-frequency noise and grid artifacts corrupt the mathematical properties of the continuous operators.
The Intuition of Gaussian Discretization
To preserve continuous symmetries on a discrete grid, researchers shifted from finite differences to Gaussian derivative discretization.
Instead of sampling discrete points directly and subtracting them, this approach leverages a elegant identity from calculus. If we want to find the derivative of a discrete signal, we can first smooth it by convolving it with a continuous Gaussian kernel $G$. Due to the derivative rule for convolutions, taking the derivative of a smoothed signal is mathematically identical to convolving the original signal with the analytically calculated derivative of the Gaussian kernel:
$$\frac{\partial}{\partial x} (f * G) = f * \left(\frac{\partial G}{\partial x}\right)$$
Gaussian Derivative (Isotropic):
_.._
.' `. Analytical derivative sampled onto a local stencil
/ /\ \ --> Smooth, isotropic, and rotationally steerable
| / \ |
By sampling the continuous, analytically derived function $\frac{\partial G}{\partial x}$ onto a local stencil, we create a convolutional kernel that acts as a robust discrete differential operator.
Why It Works: Smoothness and Isotropy
As explored by Jenner et al. in their work on steerable partial differential operators, Gaussian discretization provides two transformative advantages over finite differences:
1. Inherent Smoothing
Discrete grids are highly sensitive to high-frequency noise and aliasing. When an image or a physical field is transformed, these high-frequency artifacts corrupt standard finite difference calculations. Because the Gaussian kernel acts as a low-pass filter, it naturally suppresses noise below the grid scale, stabilizing the numerical derivatives and maintaining mathematical integrity (He et al.).
2. True Rotational Steerability
A standard Gaussian function is perfectly isotropic—it looks identical from any angle. Because its shape does not favor the grid axes, its analytical derivatives transform predictably under arbitrary rotations.
When we map these continuous derivatives to a discrete voxel or pixel grid, the isotropic nature of the underlying Gaussian mitigates the discretization errors that typically break symmetry properties (Shen et al.). This allows neural networks to achieve true steerability and equivariance for arbitrary continuous rotations, rather than being trapped in 90-degree increments.
Applications in Parameter-Efficient Deep Learning
By parameterizing convolutional filters as linear combinations of these robust, steerable Gaussian PDOs, deep learning models can achieve strict geometric equivariance with a fraction of the parameters required by traditional architectures. Instead of forcing a network to learn massive, unconstrained pixel matrices for every possible orientation, the network only needs to learn a few coefficients for the underlying differential operators.
For instance, He et al. introduced Neural ePDOs, which use a small auxiliary network to dynamically predict position-specific coefficients for these Gaussian operators based on local image features. Because the underlying Gaussian derivatives maintain their mathematical properties across continuous transformations, these models achieve state-of-the-art performance on rotation-heavy datasets like MNIST-rot while using as little as 10% of the parameters of previous top models.
By changing how we discretize, Gaussian derivative kernels allow neural networks to respect the continuous geometry of the physical world while operating entirely within the boundaries of discrete computational grids.
Key Foundational Work
Real-space grids and the Octopus code as tools for the development of new simulation approaches for electronic systems (2015) — contributed 1 concept elements to this direction.
On a Molecular Based Q-Tensor Model for Liquid Crystals with Density Variations (2015) — contributed 1 concept elements to this direction.
A code to solve the Vlasov–Fokker–Planck equation applied to particle transport in magnetic turbulence (2010) — contributed 1 concept elements to this direction.