Orthogonal Decomposition of Neural Network Kernels

70 concepts60 papers

Deep neural networks are notoriously difficult to analyze because of their non-linearities. When an input passes through a ReLU or Sigmoid layer, it undergoes a complex warping that scrambles our standard mathematical tools.

However, if we can break these non-linearities down into a sum of simpler, independent components, the mathematics becomes tractable. This is the core idea behind orthogonal decomposition in deep learning theory: treating activation functions and kernels like signals that can be decomposed into orthogonal polynomial bases. Just as Fourier analysis breaks down a jagged waveform into clean sine waves, orthogonal decomposition allows researchers to linearize the behavior of neural networks, revealing why they generalize, how fast they train, and how their geometry evolves.

The Choice of Basis: Hermite vs. Spherical Harmonics

To decompose a function effectively, we need a coordinate system—a basis—where the axes are orthogonal under the data's natural distribution. In deep learning theory, two geometric settings dominate this literature depending on how the input data is structured.

Gaussian Distributions and Hermite Polynomials

When inputs or weights follow a Gaussian distribution, Hermite polynomials provide the natural orthonormal basis. As shown by Goel et al., decomposing activations into a Hermite series allows researchers to isolate specific orders of interaction between inputs and weights.

By picking apart the non-linearity degree by degree, complex expectations simplify into clean, independent polynomial terms. This technique has been instrumental in tracking signal propagation through deep layers (Banerjee et al.) and proving tight bounds on the smallest eigenvalue of the Neural Tangent Kernel (NTK) (Nguyen et al.).

Spherical Distributions and Spherical Harmonics

When data is normalized to lie on a unit hypersphere, the rotational symmetry makes spherical harmonics (along with Legendre and Gegenbauer polynomials) the ideal basis. In a foundational work, Smola et al. demonstrated that the eigenfunctions of dot-product kernels on a sphere are exactly spherical harmonics.

This spherical perspective allows researchers to move beyond Gaussian assumptions to analyze learning problems under any rotationally invariant input distribution (Joshi et al.), mapping the network's approximation power directly to frequency components on the sphere (Daniely).

Why It Works: Diagonalization via the Funk-Hecke Theorem

The true power of this decomposition lies in its ability to diagonalize the integral operators associated with neural network kernels. In linear algebra, a matrix is easiest to understand when expressed in terms of its eigenvectors and eigenvalues, turning a coupled system of equations into independent scalar multiplications. For continuous kernels, the equivalent process is Mercer's decomposition.

When dealing with dot-product kernels on a sphere, a classical mathematical tool called the Funk-Hecke Theorem acts as a shortcut. It states that any spherical harmonic function is an eigenfunction of an integral operator defined by a dot-product kernel.

As leveraged by Vempala et al., this theorem allows researchers to compute the exact eigenvalues of a kernel in closed form based purely on the coefficients of its polynomial expansion. Instead of tracking a messy, coupled optimization process where every neuron interacts with every other neuron, training can be viewed as the independent evolution of different frequency components within a spherical harmonic basis.

Intuition: Understanding Spectral Bias

One of the most celebrated insights derived from this framework is the explanation of spectral bias—the empirical observation that neural networks learn smooth, low-frequency patterns before fitting sharp, high-frequency details.

By decomposing the NTK using spherical harmonics, Basri et al. showed that the eigenvalues associated with low-frequency spherical harmonics decay much slower than those associated with high frequencies. Because the convergence rate of gradient descent is directly proportional to these eigenvalues, the network naturally prioritizes low-frequency components.

Target Function: Low Frequency + High Frequency Noise
   |  
   v  [Training Begins]
Step 1: Network fits the dominant, low-frequency spectral components first.
   |  
   v  [Training Continues]
Step 2: Network slowly accommodates the rapidly decaying high-frequency eigenvalues.

The authors demonstrated that this spectral signature dictates the exact rate of convergence for different patterns. Geifman et al. extended this spectral analysis to convolutional architectures, proving that while CNNs share this low-frequency bias, their structural design uniquely optimizes them to learn high-frequency patterns rapidly if those patterns are spatially localized.

Geometry and the Optimization Landscape

Orthogonal decomposition does more than just predict learning speeds; it also reveals the geometric properties of hidden representations and the shape of the loss landscape.

Preventing Representation Collapse

As a network grows deeper, representations risk collapsing into a single point or a low-dimensional subspace. Joudaki et al. used Hermite expansions to derive an "isometry coefficient" for activation functions. This coefficient serves as a theoretical tool to rank activations based on their capacity to preserve input variance and resist representation collapse, ensuring that data points remain distinct and well-spread across high-dimensional space.

Guaranteeing Optimization Success

On the optimization front, Xie et al. linked the representational capacity of a network to the spectral properties of its activation's dot-product kernel, proving that sufficient neuron diversity eliminates spurious local minima.

Similarly, Karhadkar et al. utilized spherical harmonics and a hemisphere transform to establish tight lower bounds on the NTK's smallest eigenvalue. This geometric mapping guarantees that wide neural networks can effectively optimize and memorize data even in fixed-dimensional settings, providing a clear mathematical foundation for why overparameterized models converge so reliably to global optima.

Key Foundational Work

Regularization with Dot-Product Kernels (2000) — contributed 3 concept elements to this direction.
Mercer kernels and integrated variance experimental design: connections between Gaussian process regression and polynomial approximation (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spherical_harmonic_decomposition_4dd672da,
  title = {spherical harmonic decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spherical-harmonic-decomposition/rdr_4dd672da886e4036832b0ad6c1c2ff2d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spherical_harmonic_decomposition_4dd672da_v1,
  title = {spherical harmonic decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spherical-harmonic-decomposition/rdr_4dd672da886e4036832b0ad6c1c2ff2d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake