Diffusion Leaky Zero Attracting Least Mean Square Algorithm and Its Performance Analysis

Long Shi, Long Shi 0002, Haiquan ZhaoIEEE Access 2018 · DOI

TL;DR

This paper introduces two novel adaptive filtering algorithms, LZA-DLMS and LRZA-DLMS, designed to improve performance in sparse distributed systems. By incorporating sparsity-promoting penalties into a leaky diffusion least-mean-square framework, these algorithms achieve faster convergence and lower estimation error, particularly in challenging scenarios with correlated inputs and low signal-to-noise ratios.

What Problem Does This Paper Address?

Distributed estimation, where multiple nodes in a network collaboratively estimate an unknown system, is crucial for applications like environmental monitoring and source localization. Existing diffusion strategies, like the Diffusion Least-Mean-Square (DLMS), offer robustness but can struggle with systems exhibiting high eigenvalue spread and low signal-to-noise ratios (SNRs). Furthermore, many real-world systems are inherently sparse, meaning most of their parameters are zero or near-zero. Standard DLMS algorithms can perform poorly on sparse systems, and even the "leaky" DLMS, while improving stability, doesn't fully exploit sparsity. This paper aims to bridge this gap by developing algorithms that effectively handle both leaky (stable) and sparse system characteristics in a distributed setting.

Methods

The core innovation lies in modifying the cost function of the diffusion least-mean-square algorithm. The authors introduce two key modifications:

  • Leaky Zero Attracting DLMS (LZA-DLMS): This algorithm adds an $l_1$-norm penalty to the cost function. The $l_1$-norm is known to promote sparsity by driving some coefficients exactly to zero. This helps the algorithm converge faster by identifying and zeroing out irrelevant system parameters.
  • Leaky Reweighted Zero Attracting DLMS (LRZA-DLMS): To address time-varying sparsity, this algorithm replaces the $l_1$-norm penalty with a log-sum function. This "reweighted" approach is more sophisticated and can adapt to changes in the system's sparsity over time.

Both algorithms are presented in two variants: Combine-then-Adapt (CTA) and Adapt-then-Combine (ATC). The paper focuses its theoretical analysis on the ATC variants, deriving their transient and steady-state behaviors. Key analytical tools include weight error variance relations and statistical assumptions about input signals and noise.

Datasets

Not discussed in the paper. The paper uses synthetic data generated for simulations.

Key Insights and Ideas

  • Sparsity is Key: The paper strongly emphasizes that exploiting the sparsity of the unknown system is crucial for improving estimation performance in distributed adaptive filtering.
  • $l_1$-Norm for Sparsity: The $l_1$-norm penalty is a well-established technique for inducing sparsity, and its integration into the DLMS framework is a direct and effective approach.
  • Reweighting for Dynamic Sparsity: The LRZA-DLMS algorithm's use of a log-sum function for reweighting is a clever way to handle systems where sparsity changes over time, a common scenario in real-world applications.
  • ATC vs. CTA: The paper reiterates the finding that ATC-type diffusion strategies generally outperform CTA-type strategies, especially under fair comparisons.
  • Trade-offs: While the proposed algorithms excel in sparse systems, the paper notes that for non-sparse systems, the zero-attractors can sometimes be detrimental, attracting non-zero coefficients to zero.

Key Results

  • Improved Performance in Sparse Systems: Simulations demonstrate that both LZA-DLMS and LRZA-DLMS algorithms significantly outperform existing methods like leaky DLMS, especially in highly sparse scenarios.
  • LRZA-DLMS Superiority: The LRZA-DLMS algorithm generally shows better performance than LZA-DLMS, particularly when the system's sparsity changes.
  • ATC-LRZA-DLMS Best: The ATC-LRZA-DLMS algorithm is highlighted as achieving the best performance across various stages of sparsity.
  • Correlated Inputs: Even with correlated inputs and a non-sparse system, the ATC-LRZA-DLMS algorithm maintains good performance.
  • Theoretical Validation: The paper provides theoretical analysis for the transient and steady-state behaviors of the ATC-type algorithms, and simulation results show excellent agreement with these theoretical predictions.
  • Stability Bounds: The paper derives stability bounds for the step-size, ensuring reliable convergence for the proposed algorithms.

Limitations and Future Work

  • Non-sparse Systems: The paper acknowledges that the zero-attractors in the proposed algorithms might not be ideal for systems that are not sparse, as they can unnecessarily drive non-zero coefficients to zero.
  • Practical Assumptions: The theoretical analysis relies on several statistical assumptions (e.g., Gaussian inputs, i.i.d. noise) that may not always hold true in practical applications.
  • Future Work: The authors suggest exploring the use of a time-varying leaky factor to further enhance estimation performance.

Summary updated

BibTeX
@article{shi2018diffusion,
  author = {Long Shi and Long Shi and Haiquan Zhao},
  title = {{Diffusion Leaky Zero Attracting Least Mean Square Algorithm and Its Performance Analysis}},
  journal = {IEEE Access},
  year = {2018},
  doi = {10.1109/access.2018.2871555},
}
Figure