Abstract
This paper surveys the paradigm shift from traditional Cloud Machine Learning (Cloud ML), where data is transmitted to centralized models, to Embedded Machine Learning (Embedded ML), where models are transmitted to local data. As neural networks grow in complexity, communicating their parameters across distributed systems becomes a primary bottleneck. This survey provides a comprehensive taxonomy of communication settings—including on-device inference, federated learning, and distributed training—and evaluates the compression techniques, such as pruning, quantization, and sparsification, that enable efficient model exchange over bandwidth-constrained networks.
Introduction: From Cloud ML to Embedded ML
For the past decade, the dominant paradigm in artificial intelligence has been Cloud ML. In this setup, hardware-constrained devices like smartphones or IoT sensors collect raw data and transmit it to a powerful central server for processing. However, this approach faces significant challenges: shipping raw data creates substantial privacy risks, consumes massive amounts of bandwidth, and introduces a single point of failure for security.
The authors argue for a shift toward Embedded ML, characterized by "bringing the model to the data." By keeping data on the local device and communicating only neural network parametrizations (such as weights or gradients), privacy is enhanced and bandwidth usage can be optimized. The central challenge lies in the scale of modern neural networks; communicating millions or billions of parameters over mobile networks requires advanced compression strategies.
Communication Settings and Methodologies
The survey categorizes the field into four primary communication settings, each with distinct requirements for efficiency and synchronization:
- On-device Inference: Deploying pre-trained models from a server to a fleet of edge devices.
- Federated Learning (FL): A collaborative training approach where multiple clients jointly train a global model via a central server without ever sharing raw local data.
- Peer-to-Peer (P2P) Learning: A fully decentralized training architecture using gossip protocols, where devices communicate directly with neighbors rather than a central hub.
- Distributed Training: Parallelizing the training process across multiple high-performance machines within a data center to accelerate convergence.
To make these settings feasible, several compression "levers" are employed. Trained Compression involves pruning redundant weights or using knowledge distillation to train smaller "student" models. Lossy Compression techniques, such as gradient sparsification and sketching, reduce the volume of data sent during training. Furthermore, Communication Strategies like "Local SGD" allow clients to perform multiple local updates before synchronizing, significantly reducing communication frequency.
Key Insights and Technical Milestones
One of the most critical technical insights discussed is the Asymmetry of Federated Learning. In FL, the "upstream" (client-to-server) communication can often be compressed more aggressively than the "downstream" (server-to-client). Because the server aggregates updates from thousands of clients, individual quantization errors tend to cancel out, allowing for extremely low-bitrate updates without compromising the global model's integrity.
Another vital distinction is made between Biased and Unbiased Compression. While unbiased methods like Quantized SGD (QSGD) are mathematically robust, biased methods like top-k sparsification offer much higher compression rates. To prevent model divergence in biased settings, Error Accumulation is used to track quantization residuals and reintegrate them into subsequent update steps.
Quantitatively, the field has reached impressive milestones:
- Deep Compression: Combining pruning, quantization, and Huffman coding has achieved up to x49 compression on VGG models with no loss in accuracy.
- Gradient Sparsification: In data center environments, top-k sparsification combined with error accumulation can reach compression rates exceeding x500.
Challenges and Future Directions
Despite these advancements, several hurdles remain. The Non-IID (Independent and Identically Distributed) nature of edge data means that data on one device may differ wildly from another, causing standard optimization algorithms to struggle. Additionally, there is a growing Privacy-Robustness Trade-off; while FL protects data privacy, the inability of the server to inspect raw data makes it harder to detect malicious "poisoning" attacks.
Finally, the authors highlight that future research must move beyond simply reducing bit counts. On mobile devices, the energy cost of the computation required to compress data can sometimes exceed the energy saved by reducing radio transmission. Moving toward energy-aware, asynchronous, and standardized communication protocols (such as MPEG-7 Part 17) represents the next frontier for the industry.
Summary updated