Kernel methods are prized for their ability to model complex, non-linear relationships by implicitly mapping data into high-dimensional feature spaces. However, they face a fundamental "quadratic bottleneck": to compare $N$ data points, standard algorithms traditionally construct a Gram matrix of size $N \times N$. As $N$ grows into the millions, storing this matrix requires terabytes of RAM, rendering exact kernel methods impossible on standard hardware.
A growing body of research addresses this by treating the kernel matrix not as a static table to be stored, but as a dynamic operator to be computed on the fly. By shifting from explicit storage to lazy evaluation, these methods reduce memory complexity from quadratic $O(N^2)$ to linear $O(N)$ or even constant $O(1)$.
The Intuition of Lazy Evaluation
The core insight across this field is that most kernel algorithms—such as Support Vector Machines (SVMs), Gaussian Processes (GPs), and Principal Component Analysis (KPCA)—do not actually require the full Gram matrix. Instead, they interact with the data through specific operations, most notably matrix-vector multiplications ($K\mathbf{v}$).
Ament et al. et al. demonstrate this in the context of high-dimensional Bayesian Optimization. By using structured automatic differentiation, the authors perform kernel matrix-vector products lazily, allowing them to compute gradients and Hessians without ever materializing the full differentiated matrix. This mechanism ensures that memory usage remains constant relative to input dimensionality, enabling first-order optimization in spaces previously considered unreachable.
Similarly, Emanuele Sansone shows that for Positive Unlabeled (PU) learning, one can adapt the Sequential Minimal Optimization (SMO) algorithm to calculate kernel values dynamically. By replacing a global matrix cache with on-the-fly computation, the memory footprint scales linearly with the number of samples, allowing the model to handle millions of examples.
Hierarchical and Tree-Based Acceleration
When data has spatial or sequential structure, researchers can use specialized data structures to bypass the similarity matrix entirely.
For genomic data, Sonnenburg et al. et al. utilize suffix trees to index support vector weights. Instead of looking up a pre-calculated similarity in a table, the algorithm traverses the tree to compute kernel values on the fly during training. This approach eliminates the memory bottleneck for sequences, achieving massive speedups in both training and testing.
In the domain of regression, Shen et al. et al. employ KD-trees to approximate matrix-vector multiplications. By grouping nearby data points in a tree, the authors can summarize their collective influence on a query point, reducing the storage and computational burden from quadratic to linear. A similar logic applies to graphs; Herbster et al. et al. approximate large-scale graphs as trees, using a recursive algorithm to extract necessary sub-blocks of the kernel matrix without storing the full Laplacian pseudoinverse.
Block-wise and Stochastic Optimization
Another strategy for scaling is to break the optimization problem into manageable chunks, ensuring that only a small portion of the kernel space is resident in RAM at any time.
- Mini-Batching: Han et al. et al. frame spectral clustering as a stochastic optimization problem. By computing only the specific columns of the similarity matrix needed for a given mini-batch update, they achieve $O(N)$ complexity and converge to the exact solution without quadratic storage.
- Iterative Updates: For KPCA, Shi and Guo propose a covariance-free algorithm. They treat individual columns of the Gram matrix as samples for an incremental update rule, slashing space complexity to $O(N)$.
- Divide and Conquer: Graf et al. et al. introduce the Cascade SVM, which filters data through a hierarchy of smaller SVMs. This ensures that each local processor only ever handles a kernel matrix small enough to fit in its local cache.
Structural Masking and Local Expansions
Recent work has also explored how to exploit the mathematical structure of specific kernels to avoid full materialization. Lebeau et al. et al. analyze "banded" kernel matrices for data streams, where a sliding window (a Toeplitz mask) restricts similarity computations to recent samples. This allows for spectral analysis under strict memory constraints by focusing only on local data relationships.
For complex sequential signals, Tamayo-Rios et al. et al. developed PowerSig. Rather than solving a global, memory-intensive partial differential equation to compute signature kernels, the authors use local "tilewise" power series expansions. This localized computation achieves linear memory scaling, enabling the analysis of time series with over a million data points on a single GPU.
By moving away from the "compute-then-store" paradigm toward "compute-as-needed" frameworks, these diverse approaches collectively demonstrate that the power of kernel methods is not bound by the quadratic limits of physical memory.
Key Foundational Work
Large scale genomic sequence SVM classifiers (2005) — contributed 2 concept elements to this direction.
Fast Gaussian Process Regression using KD-Trees (2005) — contributed 1 concept elements to this direction.
Parallel Support Vector Machines: The Cascade SVM (2004) — contributed 1 concept elements to this direction.