LASER: Linear Compression in Wireless Distributed Optimization

Ashok Vardhan Makkuva, Marco Bondaschi, Thijs Vogels, Martin Jaggi, Hyeji Kim, Michael C. Gastpar

Introduction

In the era of large-scale distributed machine learning, the "communication bottleneck" remains a primary obstacle. Training state-of-the-art models like GPT-2 requires workers to transmit massive gradient vectors to a central server. While existing solutions address this in noiseless environments, real-world wireless and federated learning settings are often plagued by channel noise. LASER (LineAr CompreSsion in WirEless DistRibuted Optimization) is a novel gradient compression scheme designed specifically for these noisy conditions. By leveraging the low-rank nature of gradients, LASER achieves up to a 165x reduction in communication volume and requires 16x less power than standard baselines to reach equivalent model performance.

The Challenge of Noisy Wireless Channels

Standard distributed training often relies on error-correcting codes to handle noisy channels. However, these codes introduce significant latency, as the server must wait for perfect decoding from every worker before aggregation. While "over-the-air" computation allows for direct signal aggregation in the physical layer, it typically struggles to scale with the high dimensionality of deep neural networks. LASER bridges this gap by providing a scalable, mathematically robust compression method tailored for over-the-air aggregation.

The LASER Methodology

LASER's core innovation lies in its approach to Linear Compression. It operates on the observation that gradients in deep learning are often approximately low-rank. The method follows four key steps:

  1. Low-Rank Decomposition: Using PowerSGD, LASER decomposes the high-dimensional gradient matrix $g$ into two smaller factors, $p$ and $q$ (where $g \approx pq^\top$).
  2. Optimal Power Allocation: LASER employs a principled power-allocation strategy to maximize the Signal-to-Noise Ratio (SNR) for these factors, minimizing reconstruction error at the server.
  3. Over-the-Air Aggregation: By exploiting the physical properties of wireless channels, signals from multiple workers sum naturally in the air, enabling "native" gradient averaging.
  4. Error Feedback (EF): To maintain convergence despite lossy compression, LASER utilizes an error-feedback mechanism that "remembers" compressed-away gradient components and reintroduces them in subsequent steps.

Experimental Evaluation

The authors benchmarked LASER against several state-of-the-art schemes, including Sketching, Signum, and Random-K, across diverse tasks ranging from language modeling (GPT-2) to image classification (ResNet18).

Perplexity vs. Power Budget for GPT-2 on WikiText-103. LASER consistently achieves lower perplexity than baselines across all power regimes.

Key Insights and Results

A critical insight of the LASER framework is the "SNR Boost." By transmitting a compressed rank-$r$ version of the gradient, workers can allocate more power per coordinate, effectively boosting the SNR by a factor proportional to the matrix dimension. This makes LASER particularly effective in high-noise environments.

Furthermore, the researchers discovered that a constant power policy—maintaining the same transmission power throughout training—outperforms dynamic schemes. As the model converges and gradients shrink, the effective SNR implicitly improves over time.

Power required to reach target perplexity levels for GPT-2. LASER demonstrates a consistent 16x reduction in power compared to Z-SGD.

The results demonstrate LASER's efficiency:

  • GPT-2 Performance: LASER achieved a 50-64% improvement in perplexity over baselines in low-to-moderate power regimes.
  • Power Efficiency: As shown in the table above, LASER requires 16x less power than Z-SGD to reach target perplexity levels.
  • Communication Volume: Data sent per iteration was reduced by 165x (from 496 MB to 3 MB) compared to uncompressed gradients.

Conclusion and Future Directions

LASER provides a robust framework for wireless distributed optimization, proving that nearly the same convergence rates as classical SGD can be achieved with a significantly lower noise penalty. Future work may address downlink noise in peer-to-peer settings, eliminate the need for Channel State Information (CSI) in fast-fading environments, and extend compression to bias vectors for even greater efficiency.

Summary updated

Abstract

Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels.
BibTeX
@inproceedings{makkuva2024laser,
  author = {Ashok Vardhan Makkuva and Marco Bondaschi and Thijs Vogels and Martin Jaggi and Hyeji Kim and Michael C. Gastpar},
  title = {{LASER: Linear Compression in Wireless Distributed Optimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=sDjszMb2Ir},
}
Figure