Deep Spectral Kernel Learning

Hui Xue, Zheng-Fan Wu, Wei-Xiang Sun

Abstract

This paper introduces the Deep Spectral Kernel Network (DSKN), a novel architecture that stacks non-stationary and non-monotonic spectral kernels into a deep, hierarchical structure. By combining the mathematical rigor of kernel methods with the representational efficiency of deep learning, DSKN captures complex, long-range data correlations that traditional "local" kernels and standard shallow models often miss.

Introduction

Traditional kernel methods, such as the Gaussian RBF kernel, are often limited by their locality. They are typically stationary (similarity depends only on distance) and monotonic (similarity decreases as distance increases). These properties hinder their ability to model complex environments where patterns may repeat over long distances or vary across the input space.

While spectral kernels were developed to address these limitations using Fourier analysis, they are generally "shallow" models. To represent complex functions, shallow models require an exponential number of computational elements. Conversely, Deep Neural Networks (DNNs) offer representational efficiency but are often viewed as "black boxes" lacking a rigorous theoretical framework. DSKN aims to bridge this gap by creating a deep architecture where every layer is explicitly a spectral kernel mapping.

Methodology: Deep Spectral Kernel Networks

The DSKN architecture is built on several core technical innovations:

  1. Generalized Spectral Form: Utilizing Yaglom’s theorem, the authors derive a general form for non-stationary spectral kernels that can adapt to different regions of the input space.
  2. Monte Carlo Approximation: The kernel mapping is approximated using a set of cosine functions with learnable frequency matrices ($\Omega, \Omega'$). This transforms the kernel into a functional mapping $\Phi(x)$ resembling a single-layer neural network with cosine activation.
  3. Hierarchical Stacking: These mappings are stacked such that the output of one layer becomes the input for the next: $K^{(l)}(x, x') = \Xi^{(l)}(x) \cdot \Xi^{(l)}(x')$. This hierarchy allows the model to represent complex, high-level concepts more compactly than shallow kernels.
  4. Unified Spectral Transform: A technique to initialize the network by transforming domain-specific prior knowledge (e.g., Gaussian or Laplacian priors) into the initial frequencies of the DSKN.
The hierarchical architecture of DSKN, showing the stacking of spectral kernel layers.

The entire network is end-to-end trainable using standard backpropagation, where the learned "weights" are the spectral frequencies of the kernels. This provides a theoretical grounding in Reproducing Kernel Hilbert Space (RKHS) theory, including a recursive Representer Theorem.

Experimental Evaluation

The DSKN was evaluated against several benchmarks, including UCI classification tasks (four, ionosphere, splice, wbdc), regression tasks (airfoil, boston, concrete, energy), and the MNIST image classification dataset.

Key Results

  • Superior Performance: DSKN consistently outperformed 1-layer spectral kernels (1-SKN), standard DNNs, and hybrid models like Deep Kernel Learning (DKL) across nearly all tasks.
  • Robustness to Data Scarcity: A significant advantage of DSKN is its performance under limited training data. On the MNIST dataset, DSKN maintained high accuracy even at low sampling rates, outperforming standard DNNs. For instance, at a 5% training scale, DSKN achieved 97.62% accuracy compared to the DNN's 95.89%.
Accuracy vs. Sampling Rate on MNIST, demonstrating DSKN's superior performance in data-sparse regimes.

Discussion and Future Work

DSKN provides a powerful inductive bias for small-to-medium scale tasks by embedding spectral kernel properties directly into a deep architecture. Unlike "Deep Kernel Learning" (DKL), which often uses a DNN as a feature extractor before a standard kernel, DSKN integrates the hierarchy within the kernel mappings themselves.

Limitations: The recursive nature of deep kernels can be computationally demanding compared to highly optimized CNNs on massive datasets. Future research may focus on automated architecture search for optimal depth and width, and exploring how different "base" kernels (like Cauchy or Laplacian) interact when stacked for specialized applications in signal processing or physics.

Summary updated

Abstract

Recently, spectral kernels have attracted wide attention in complex dynamic environments. These advanced kernels mainly focus on breaking through the crucial limitation on locality, that is, the stationarity and the monotonicity. But actually, owing to the inefficiency of shallow models in computational elements, they are more likely unable to accurately reveal dynamic and potential variations. In this paper, we propose a novel deep spectral kernel network (DSKN) to naturally integrate non-stationary and non-monotonic spectral kernels into elegant deep architectures in an interpretable way, which can be further generalized to cover most kernels. Concretely, we firstly deal with the general form of spectral kernels by the inverse Fourier transform. Secondly, DSKN is constructed by embedding the preeminent spectral kernels into each layer to boost the efficiency in computational elements, which can effectively reveal the dynamic input-dependent characteristics and potential long-range correlations by compactly representing complex advanced concepts. Thirdly, detailed analyses of DSKN are presented. Owing to its universality, we propose a unified spectral transform technique to flexibly extend and reasonably initialize domain-related DSKN. Furthermore, the representer theorem of DSKN is given. Systematical experiments demonstrate the superiority of DSKN compared to state-of-the-art relevant algorithms on varieties of standard real-world tasks.
BibTeX
@inproceedings{xue2019deep,
  author = {Hui Xue and Zheng-Fan Wu and Wei-Xiang Sun},
  title = {{Deep Spectral Kernel Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/558},
}
Figure