BaKer-Nets: Bayesian Random Kernel Mapping Networks

Hui Xue, Zheng-Fan Wu

Introduction

Deep Spectral Kernel Networks (DSKNs) represent a significant advancement in kernel methods, moving beyond the limitations of local kernels by employing periodic computational elements like sine and cosine functions. This periodicity allows DSKNs to capture long-range dependencies and complex, non-stationary patterns across the entire feature space. However, this same power introduces a significant optimization challenge known as the "cask effect." The periodic nature of the basis functions creates a "rugged" loss landscape characterized by a high density of poor local minima. Consequently, standard gradient-based optimization often becomes trapped in suboptimal solutions, preventing DSKNs from reaching their theoretical performance potential.

Methodology

To address the optimization difficulties inherent in DSKNs, we introduce Bayesian random Kernel mapping Networks (BaKer-Nets). This framework transforms the optimization problem into a probabilistic search by treating network weights as random variables rather than fixed points.

The Copula-Based Bridge

The core of the BaKer-Net architecture is the use of Copulas to model weight uncertainty. Copulas are mathematical tools that allow for the construction of complex joint distributions by joining simpler marginal distributions. In BaKer-Nets, they serve as a bridge between classic kernel knowledge (the prior) and the learned network parameters (the posterior). This approach enables the model to capture intricate correlations between weights that standard Gaussian assumptions might miss.

The BaKer-Net framework utilizing a Copula-based bridge between prior and posterior distributions.

Bayesian Learning Paradigm

Training is conducted using Stochastic Variational Inference (SVI) to maximize the Evidence Lower Bound (ELBO). This objective function naturally balances "Structural Risk"—the complexity of the model relative to the prior—and "Empirical Risk"—the fit to the observed data. By sampling weights from the learned distributions during training, the model effectively explores multiple potential solutions simultaneously, acting as an "infinite ensemble" that can navigate through the traps of a rugged loss landscape.

Experimental Evaluation

BaKer-Nets were evaluated against standard Deep Neural Networks (DNNs), Deep Kernel Learning (DKL), and conventional DSKNs across various benchmarks, including UCI/LIBSVM datasets and image classification tasks (MNIST, Fashion-MNIST, and CIFAR-10).

Performance and Convergence

BaKer-Nets consistently achieved superior accuracy or lower Mean Squared Error (MSE) across nearly all tested datasets. Notably, on the CIFAR-10 dataset, BaKer-Net reached a convergent accuracy of 0.674, significantly outperforming standard DSKNs (0.549) and Spectral Random Fourier Features (SRFF, 0.529).

The primary advantage of the Bayesian approach is its ability to escape local minima. While standard DSKNs often stall in shallow basins, the uncertainty-driven exploration in BaKer-Nets allows the model to "jump" out of these traps to find deeper, more robust global minima.

Visualization of the learning process on a rugged loss landscape.

Conclusion

BaKer-Nets demonstrate that incorporating well-tuned uncertainty via a Bayesian framework is a powerful tool for optimizing complex kernel networks. By smoothing the loss landscape and enabling probabilistic exploration, BaKer-Nets overcome the "cask effect" that plagues traditional DSKNs. Future work will focus on reducing the computational overhead associated with Monte Carlo sampling and exploring the scalability of this approach to larger architectures like Transformers.

Summary updated

Abstract

Recently, deep spectral kernel networks (DSKNs) have attracted wide attention. They consist of periodic computational elements that can be activated across the whole feature spaces. In theory, DSKNs have the potential to reveal input-dependent and long-range characteristics, and thus are expected to perform more competitive than prevailing networks. But in practice, they are still unable to achieve the desired effects. The structural superiority of DSKNs comes at the cost of the difficult optimization. The periodicity of computational elements leads to many poor and dense local minima in loss landscapes. DSKNs are more likely stuck in these local minima, and perform worse than expected. Hence, in this paper, we propose the novel Bayesian random Kernel mapping Networks (BaKer-Nets) with preferable learning processes by escaping randomly from most local minima. Specifically, BaKer-Nets consist of two core components: 1) a prior-posterior bridge is derived to enable the uncertainty of computational elements reasonably; 2) a Bayesian learning paradigm is presented to optimize the prior-posterior bridge efficiently. With the well-tuned uncertainty, BaKer-Nets can not only explore more potential solutions to avoid local minima, but also exploit these ensemble solutions to strengthen their robustness. Systematical experiments demonstrate the significance of BaKer-Nets in improving learning processes on the premise of preserving the structural superiority.
BibTeX
@inproceedings{xue2020baker,
  author = {Hui Xue and Zheng-Fan Wu},
  title = {{BaKer-Nets: Bayesian Random Kernel Mapping Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/425},
}
Figure