Abstract: The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted as an approximate Bayesian filtering method. This allows us to systematically develop extensions of the KLMS by modifying the underlying state-space and observation models. The resulting extensions introduce many desirable properties such as "forgetting", and the ability to learn from discrete data, while retaining the computational simplicity and time complexity of the original algorithm.
Introduction
Adaptive filtering is a cornerstone of real-time learning in dynamic environments. While the Kernel Least Mean Squares (KLMS) algorithm is widely used for its computational efficiency and ability to handle nonlinearities, it faces significant challenges: memory growth proportional to data size, a narrow focus on continuous regression, and a reliance on heuristic "hacks" for tracking moving targets. This work provides a formal Bayesian foundation for KLMS, revealing it as an approximate Kalman filter. By adopting a state-space perspective, we derive principled extensions that address these limitations, enabling efficient "forgetting" and adaptation to discrete data types.
Methodology: KLMS as Bayesian Filtering
The core contribution of this research is the re-derivation of KLMS from a Bayesian filtering perspective. The standard KLMS update rule is shown to be mathematically equivalent to finding the Maximum A Posteriori (MAP) estimate of a state-space model where weights undergo a random walk (diffusion).
To maintain the algorithm's $O(n)$ complexity, the authors employ approximate inference, assuming the posterior distribution is a delta function at the MAP estimate. This effectively discards the covariance (uncertainty) at each step, which is the key to the algorithm's speed. This insight clarifies that the constant learning rate in standard KLMS corresponds to the ratio between diffusion variance and noise variance in a non-stationary model.
Principled Extensions
Leveraging this Bayesian framework, several extensions are proposed:
- fKLMS (Forgetful KLMS): By replacing the random walk with an Ornstein-Uhlenbeck process, a leakage factor is introduced. This causes old observations to decay exponentially, allowing for principled model pruning and tracking in non-stationary environments.
- Generalized Observation Models: The framework allows swapping the standard Gaussian likelihood for Poisson (for count data) or Bernoulli (for binary classification) likelihoods. Even with these non-linear models, the update remains a simple one-dimensional scalar optimization at each time step.
Experimental Evaluation
The proposed extensions were evaluated across diverse scenarios, including synthetic non-stationary Gaussian processes, neural simulations of drifting tuning curves, and real-world wireless communication data featuring nonlinear Rayleigh fading.
Key results include:
- Superior Tracking: In non-stationary tasks, fKLMS outperformed standard KLMS and Quantized KLMS (QKLMS), achieving a significantly lower asymptotic normalized mean squared error (-3.93 dB vs -3.33 dB).
- Versatility: The Poisson and Logistic extensions successfully tracked shifting distributions that standard regression-based KLMS cannot handle.
- Hardware Validation: In wireless testbed experiments, fKLMS showed high performance, significantly outperforming traditional linear methods like NLMS.
Conclusion and Future Work
By framing KLMS as an approximate Bayesian filter, this work transforms ad-hoc heuristics into principled mathematical operations. While the current approach discards weight covariance to maintain efficiency—precluding the generation of confidence intervals—it provides a robust path for developing compact, adaptive nonlinear filters. Future research will explore automated hyperparameter tuning via Expectation-Maximization and formal convergence proofs for the discrete-data variants.
Summary updated