Abstract
Traditional non-parametric methods often struggle with high-dimensional data due to the "curse of dimensionality," while neural networks, despite their feature-learning capabilities, lack theoretical transparency and optimization stability. This paper introduces the Brownian Kernel Neural Network (BKerNN), a hybrid framework that integrates the structural advantages of kernel methods with the adaptive feature learning of neural networks. By utilizing the positive 1-homogeneity of the Brownian kernel and employing novel concave regularization, BKerNN achieves robust feature selection and function estimation. Empirical results across synthetic and real-world datasets demonstrate that BKerNN outperforms standard kernel ridge regression and matches or exceeds the performance of ReLU neural networks with significantly fewer parameters.
1. Introduction
In high-dimensional settings, many target functions depend only on a few relevant linear combinations of the input—a structure described by the multiple-index model. Standard Kernel Ridge Regression (KRR) relies on fixed, pre-specified features, making it susceptible to the curse of dimensionality. Conversely, neural networks (NNs) can learn these features automatically but are notoriously difficult to analyze theoretically and are prone to getting trapped in poor local minima during optimization.
The Brownian Kernel Neural Network (BKerNN) is proposed to bridge this gap. It functions as an infinite-width, one-hidden-layer network where the activation function is effectively learned through a kernel framework. This approach retains the optimization benefits of kernel methods while providing the flexibility of NNs to adapt to the underlying data structure.
2. The Brownian Kernel and Optimization
A central innovation of BKerNN is the use of the Brownian kernel: $$k^{(B)}(a, b) = \min(|a|, |b|) \mathbf{1}_{ab>0}$$
Unlike Gaussian or exponential kernels, the Brownian kernel exhibits positive 1-homogeneity. This property is critical for smoothing the optimization landscape, making the model significantly more robust to the local minima that typically hinder kernel-learning methods.
The model employs a hybrid optimization strategy that separates the learning of the output layer from the feature weights:
- Output Layer ((\alpha)): Solved in closed form, similar to traditional KRR.
- Input Layer ((w)): Updated via proximal gradient descent to learn the "features" or directions of importance.
This alternating strategy avoids the "two-timescale" training problem common in standard NNs, where the simultaneous updating of inner and outer layers can lead to training instability.
3. Advanced Regularization and Feature Selection
To facilitate feature learning in high-dimensional spaces, the authors introduce concave feature penalties, such as log-sum versions of the nuclear norm. While convex penalties like the Lasso are common, the authors demonstrate that concave versions are more effective at "driving the solution toward the boundary," thereby identifying the underlying low-rank feature subspace more accurately.
Statistical analysis using Rademacher complexity shows that BKerNN’s expected risk converges at rates of (O(\min((d/n)^{1/2}, n^{-1/6}))). Crucially, these rates suggest that BKerNN can achieve convergence that is largely independent of the input dimension (d), justifying the model's use in extreme high-dimensional scenarios where (d \gg n).
4. Empirical Performance
The BKerNN was evaluated against 17 OpenML benchmark datasets and various synthetic tasks designed to test feature recovery.
4.1. Efficiency and Robustness
In synthetic tests, BKerNN demonstrated remarkable parameter efficiency, capturing complex 1D functions with a single "particle" (neuron), whereas a standard ReLU NN required 32 neurons for a comparable fit. Furthermore, BKerNN consistently reached global minima in scenarios where models using non-homogeneous kernels (such as Exponential or Gaussian kernels) frequently failed.
4.2. Real-World Benchmarks
On the OpenML regression suite—including datasets like "Ailerons," "Diamonds," and "Houses"—BKerNN with concave regularization frequently emerged as the top performer. It consistently outperformed standard Brownian KRR and showed competitive, often superior, performance compared to standard ReLU neural networks. Notably, BKerNN showed a superior ability to identify the true underlying "relevant" features compared to standard NNs, which often failed to recover the feature subspace even as sample sizes increased.
5. Limitations and Future Work
Despite its strengths, BKerNN faces several challenges:
- Scalability: The (O(n^3)) complexity of the matrix inversion in the output layer limits its application to very large datasets without approximation methods like the Nyström method.
- Differentiability: The Brownian kernel's non-differentiability at zero complicates formal global convergence proofs, requiring proximal methods for optimization.
- Architecture Depth: The current work is limited to one-hidden-layer (shallow) architectures.
Future research will focus on extending these hybrid principles to deeper architectures and refining the theoretical bounds for the highly performant concave penalties.
Summary updated