Recursive Kernel Composition for Deep Architectures

24 concepts22 papers1 proposal

For years, machine learning was divided into two camps. On one side, kernel methods like Support Vector Machines offered mathematical elegance and convex optimization but were often seen as "shallow." On the other, deep neural networks provided powerful hierarchical representations but relied on non-convex heuristics. Recursive kernel composition bridges this gap by defining deep architectures as a sequence of kernel transformations, where the output of one layer’s kernel serves as the input for the next.

The Geometry of Depth

The core intuition behind this approach is that the hierarchical structure of a neural network can be encapsulated within a single, complex kernel function. As Cho et al. argue, recursively composing kernel functions allows kernel machines to simulate the depth of neural networks while maintaining the stability of classical learning theory.

In a standard shallow kernel, we measure the similarity between two points in a fixed feature space. In a recursive kernel, the similarity at layer $L$ depends on the similarity calculated at layer $L-1$. Poggio et al. propose that this recursive derivation allows us to ground the representational power of deep networks in formal kernel principles, effectively building a "mathematical visual cortex."

Covariance Propagation in Infinite Networks

The most significant theoretical breakthrough in this area involves the "infinite-width limit." As neural networks become infinitely wide, they converge in distribution to Gaussian Processes (GPs). The behavior of these infinite networks is entirely determined by their covariance function, or kernel.

The authors of Matthews et al. rigorously proved that for deep fully connected networks, this kernel is derived through an analytic transformation of the previous layer's covariance. For common activations like ReLU, Zhao et al. explain that these transformations involve specific trigonometric functions that define how the kernel evolves as information propagates deeper into the network.

This recursive framework isn't limited to simple feedforward layers:

  • Convolutional Networks: Garriga-Alonso et al. demonstrate that wide CNNs can be treated as "shallow" GPs using a recursive kernel that captures spatial dependencies. By using correlated weight priors, Garriga-Alonso et al. further show how to propagate these spatial correlations through successive layers.
  • Graph and Sequence Data: Tieu et al. use recursive computation to simulate multi-layer perceptrons for graph data, while Lei et al. derive recurrent architectures directly from string and graph kernels.
  • Neural Operators: For complex tasks like solving PDEs, de Souza et al. show that the total covariance of a deep neural operator is determined by the recursive composition of the covariance functions of its constituent layers.

Stability and Complexity

By framing deep learning as a sequence of kernel operations, researchers can analyze properties that are otherwise difficult to quantify. Bietti et al. use the "multilayer convolutional kernel" framework to prove that deep CNNs are naturally stable to small deformations and near-invariant to translations. This is achieved by modeling layers as a sequence of patch extractions and non-linear mappings into a Reproducing Kernel Hilbert Space (RKHS).

Furthermore, this perspective offers computational advantages. Lee et al. demonstrate that by shifting the goal from finding exact weights to finding a predictor in a recursively constructed kernel space, multi-layer networks can be "improperly learned" in polynomial time, bypassing the NP-hardness of traditional training.

Evolving and Stochastic Kernels

Recent work has pushed the boundaries of static kernels. Chen introduces Neural Hilbert Ladders, which model deep networks as a hierarchy of evolving function spaces. In this framework, each layer's kernel is recursively defined using the functions learned in the preceding layer, capturing how networks adapt their features during training.

Taking this a step further, Loria et al. explore networks with heavy-tailed weight priors. They find that these networks converge to $\alpha$-stable processes characterized by stochastic kernels. In these models, the covariance matrix itself becomes random and evolves through the layers, allowing the kernel to perform representation learning even in the infinite-width limit.

Through recursive composition, the "black box" of deep learning becomes a structured hierarchy of kernel transformations, providing a unified language for understanding approximation, stability, and uncertainty across diverse architectures.

Key Foundational Work

Hierarchical Learning Machines and Neuroscience of Visual Cortex (2010) — contributed 2 concept elements to this direction.
Kernel Methods for Deep Learning (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_recursive_kernel_composition_ad9bfe03,
  title = {recursive kernel composition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-kernel-composition/rdr_ad9bfe03072943198ad59fddeea2b167},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_recursive_kernel_composition_ad9bfe03_v1,
  title = {recursive kernel composition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-kernel-composition/rdr_ad9bfe03072943198ad59fddeea2b167/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake