In Federated Learning (FL), edge devices like smartphones or satellites act as decentralized researchers. They train models on private local data and send the resulting gradients back to a central server. However, modern neural networks are massive, and transmitting these high-dimensional updates over wireless connections is often described as trying to push a grand piano through a garden hose.
To make FL viable, researchers are developing techniques to shrink these updates without losing the "knowledge" they contain. This is achieved by exploiting the mathematical structure of neural network updates and the physical properties of communication channels.
Exploiting the Intrinsic Dimension
A key intuition in modern compression is that while a model might have millions of parameters, the "useful" changes during training often happen in a much smaller subspace. Luke Melas-Kyriazi and Franklyn Wang demonstrate this in Intrinsic Gradient Compression for Federated Learning. They show that massive models like BERT can be trained by communicating updates within tiny, randomly generated subspaces. By only sending the coordinates within this small subspace rather than the full parameter vector, communication costs can drop by three orders of magnitude.
Similarly, Diego Klabjan and Haemin Park propose FedLoRU, which constrains updates to a low-rank matrix. They identify a "rank paradox": forcing updates to be low-rank doesn't just save bandwidth; it actually improves performance by forcing the gradients from different clients to align more closely, acting as a form of implicit regularization.
Precision versus Accuracy
If we cannot reduce the number of parameters, we can reduce the precision of each parameter. Standard training uses 32-bit floats, but much of that detail is noise.
Extreme quantization, such as binarization, reduces each parameter to a single bit. In FedBAT: Communication-Efficient Federated Learning via Learnable Binarization, Shiwei Li argues that compression shouldn't be a post-processing step. Instead, by making the model "binarization-aware" during local training, the device learns updates that are inherently efficient to transmit.
However, aggressive compression often introduces bias, especially when data is heterogeneous (Non-IID). Zhiwei Tang and Tsung-Hui Chang address this in z-SignFedAvg. They inject a specific distribution of noise before compressing gradients to their sign (1 or -1). This "dithering" effect ensures that the compressed update remains an unbiased estimator of the true gradient, allowing 1-bit algorithms to converge as reliably as full-precision ones.
Correcting Errors with Feedback
When we compress a gradient, we lose information. If we simply discard that "residual" error, the model eventually drifts off course. Ping Li and Xiaoyun Li analyze a solution called Error Feedback (EF) in Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression.
The intuition is simple: if a client wants to send an update of 0.9 but the compressor only allows sending 1.0, the client stores the -0.1 error locally. In the next round, it adds this "stale" error to the new gradient before compressing again. This ensures that, over several rounds, the total information transmitted matches the intended update, enabling the use of highly biased compressors like Top-K sparsification.
Computing Over the Air
In wireless environments, the communication medium itself can be used as a computer. Traditional FL requires each client to have its own "slot" in the radio spectrum to avoid interference. Xiaojun Yuan proposes OA-FMTL, which uses "Over-the-Air" (OtA) computation.
Instead of avoiding interference, OtA encourages it. When multiple devices transmit their gradients simultaneously on the same frequency, the signals superimpose at the receiver. The physics of the wireless channel effectively performs the "sum" operation required for gradient averaging. By combining this with compressed sensing, the server can aggregate updates from many tasks at once, effectively multiplying the available bandwidth by the number of participating devices.