Performance Analysis of Fractional Learning Algorithms

Abdul Wahab, Shujaat Khan, Imran Naseem, Jong Chul Ye

Abstract

Fractional learning algorithms (FLAs) have recently emerged as a trending topic in signal processing and adaptive filtering, with many researchers claiming they offer superior performance over conventional integer-order methods. However, the theoretical foundations of these claims have remained largely unexamined. This article provides a rigorous mathematical analysis of fractional variants of the Least Mean Squares (LMS) and steepest descent algorithms. We identify critical "schematic kinks" in current FLA implementations, discuss their origins and consequences, and propose mathematically sound remedies. Through numerical experiments in stochastic environments, we demonstrate that the proclaimed superiority of FLAs is often a myth, as they frequently suffer from higher computational costs and stability issues without providing tangible performance gains.

Introduction

In the pursuit of faster convergence and higher precision in adaptive filtering tasks—such as noise cancellation and system identification—researchers have increasingly turned to fractional calculus. By replacing standard integer-order gradients with fractional-order derivatives, the objective was to discover "shortcuts" across the error surface to reach optimal solutions more efficiently.

Despite the proliferation of literature on the subject, the field has outpaced its theoretical grounding. Many proposed fractional LMS variants are developed heuristically, leading to a lack of rigorous performance analysis. This paper addresses this gap, investigating whether these algorithms are mathematically robust or if their perceived advantages are the result of flawed interpretations of fractional geometry.

Theoretical Foundations and the "Chain Rule Trap"

A primary finding of this analysis is the widespread misapplication of calculus in FLA literature. Most existing papers incorrectly apply the standard chain rule to fractional derivatives. In reality, the fractional chain rule—governed by the complex Faà di Bruno formula—does not simplify into the elegant update rules commonly cited.

By re-deriving these update rules using Riemann-Liouville Fractional Calculus, we reveal that the simplified "fractional gradients" used in practice do not actually point toward the true minimum (the Wiener solution). Instead, they point toward "fractional critical points," which are mathematically distinct from the optimal solution. This misalignment fundamentally undermines the algorithm's ability to converge accurately.

Identifying Schematic Kinks

Our mathematical autopsy identified several "schematic kinks" that hinder the performance of FLAs:

  1. The Complex Number Problem: Standard FLAs often encounter negative values during iteration. Taking a fractional power (e.g., $x^{0.5}$) of a negative iterate results in a complex number. This is a fundamental consequence of applying the derivative outside its domain of definition, leading to algorithm failure unless specific, often unacknowledged, mathematical fixes are applied.
  2. Steady-State Error: Unlike integer-order derivatives, the fractional derivative of a constant is not zero. Consequently, even when an algorithm reaches the true minimum, the fractional term continues to "push" the weights, leading to persistent steady-state error and preventing the algorithm from settling.
  3. Divergence in Negative Systems: Without specific remedies—such as using absolute values and switching between "left" and "right" fractional derivatives—FLAs fail to identify systems with negative weights, leading to immediate divergence.

Experimental Methodology

To evaluate these findings, we conducted system identification tasks under controlled stochastic environments. We tested three primary scenarios:

  • Systems with strictly negative weights.
  • Systems with strictly positive weights.
  • Systems with random weights.

Input signals consisted of 1,000 samples generated from a zero-mean Gaussian distribution with unit variance. To simulate realistic conditions, additive Gaussian noise was introduced to maintain a 10dB Signal-to-Noise Ratio (SNR).

Results and Performance Analysis

The numerical experiments yielded several key insights:

  • No Performance Gain: In head-to-head comparisons with standard LMS, fractional variants showed no noticeable improvement in convergence speed or final accuracy.
  • Computational Overhead: FLAs are significantly more expensive to implement. They require additional operations to calculate fractional powers and Gamma functions at every iteration, increasing the hardware and time requirements.
  • Slow Convergence in Short-Memory Variants: For certain "short-memory" fractional implementations, the convergence rate was found to be substantially slower than that of the standard LMS algorithm.

Conclusion

While fractional calculus offers a fascinating mathematical framework, its current application in learning algorithms appears to be a complex solution in search of a problem. The "schematic kinks" identified in this study—ranging from invalid calculus to gradient misalignment—suggest that FLAs, in their current form, do not outperform classical integer-order methods. Future work must focus on a deeper understanding of fractional geometry and a full stochastic study of update distributions before these algorithms can be considered a viable alternative to the robust and efficient standard LMS.

Summary updated

Abstract

Fractional learning algorithms are trending in signal processing and adaptive filtering recently. However, it is unclear whether the proclaimed superiority over conventional algorithms is well-grounded or is a myth as their performance has never been extensively analyzed. In this article, a rigorous analysis of fractional variants of the least mean squares and steepest descent algorithms is performed. Some critical schematic kinks in fractional learning algorithms are identified. Their origins and consequences on the performance of the learning algorithms are discussed and swift ready-witted remedies are proposed. Apposite numerical experiments are conducted to discuss the convergence and efficiency of the fractional learning algorithms in stochastic environments.
BibTeX
@article{wahab2022performance,
  author = {Abdul Wahab and Shujaat Khan and Imran Naseem and Jong Chul Ye},
  title = {{Performance Analysis of Fractional Learning Algorithms}},
  journal = {IEEE Trans. Signal Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=JQ0gE46DUf},
}

Loading related content...

Figure