Trends and Advancements in Deep Neural Network Communication

Felix Sattler, Thomas Wiegand, Wojciech SamekarXiv 2020 · arXiv

Abstract

This paper surveys the paradigm shift from traditional Cloud Machine Learning (Cloud ML), where data is transmitted to centralized models, to Embedded Machine Learning (Embedded ML), where models are transmitted to local data. As neural networks grow in complexity, communicating their parameters across distributed systems becomes a primary bottleneck. This survey provides a comprehensive taxonomy of communication settings—including on-device inference, federated learning, and distributed training—and evaluates the compression techniques, such as pruning, quantization, and sparsification, that enable efficient model exchange over bandwidth-constrained networks.

Introduction: From Cloud ML to Embedded ML

For the past decade, the dominant paradigm in artificial intelligence has been Cloud ML. In this setup, hardware-constrained devices like smartphones or IoT sensors collect raw data and transmit it to a powerful central server for processing. However, this approach faces significant challenges: shipping raw data creates substantial privacy risks, consumes massive amounts of bandwidth, and introduces a single point of failure for security.

Comparison of Cloud ML and Embedded ML paradigms.

The authors argue for a shift toward Embedded ML, characterized by "bringing the model to the data." By keeping data on the local device and communicating only neural network parametrizations (such as weights or gradients), privacy is enhanced and bandwidth usage can be optimized. The central challenge lies in the scale of modern neural networks; communicating millions or billions of parameters over mobile networks requires advanced compression strategies.

Communication Settings and Methodologies

The survey categorizes the field into four primary communication settings, each with distinct requirements for efficiency and synchronization:

  1. On-device Inference: Deploying pre-trained models from a server to a fleet of edge devices.
  2. Federated Learning (FL): A collaborative training approach where multiple clients jointly train a global model via a central server without ever sharing raw local data.
  3. Peer-to-Peer (P2P) Learning: A fully decentralized training architecture using gossip protocols, where devices communicate directly with neighbors rather than a central hub.
  4. Distributed Training: Parallelizing the training process across multiple high-performance machines within a data center to accelerate convergence.
The Federated Learning architecture where clients communicate model updates to a central server.

To make these settings feasible, several compression "levers" are employed. Trained Compression involves pruning redundant weights or using knowledge distillation to train smaller "student" models. Lossy Compression techniques, such as gradient sparsification and sketching, reduce the volume of data sent during training. Furthermore, Communication Strategies like "Local SGD" allow clients to perform multiple local updates before synchronizing, significantly reducing communication frequency.

Key Insights and Technical Milestones

One of the most critical technical insights discussed is the Asymmetry of Federated Learning. In FL, the "upstream" (client-to-server) communication can often be compressed more aggressively than the "downstream" (server-to-client). Because the server aggregates updates from thousands of clients, individual quantization errors tend to cancel out, allowing for extremely low-bitrate updates without compromising the global model's integrity.

Another vital distinction is made between Biased and Unbiased Compression. While unbiased methods like Quantized SGD (QSGD) are mathematically robust, biased methods like top-k sparsification offer much higher compression rates. To prevent model divergence in biased settings, Error Accumulation is used to track quantization residuals and reintegrate them into subsequent update steps.

Quantitatively, the field has reached impressive milestones:

  • Deep Compression: Combining pruning, quantization, and Huffman coding has achieved up to x49 compression on VGG models with no loss in accuracy.
  • Gradient Sparsification: In data center environments, top-k sparsification combined with error accumulation can reach compression rates exceeding x500.

Challenges and Future Directions

Despite these advancements, several hurdles remain. The Non-IID (Independent and Identically Distributed) nature of edge data means that data on one device may differ wildly from another, causing standard optimization algorithms to struggle. Additionally, there is a growing Privacy-Robustness Trade-off; while FL protects data privacy, the inability of the server to inspect raw data makes it harder to detect malicious "poisoning" attacks.

Finally, the authors highlight that future research must move beyond simply reducing bit counts. On mobile devices, the energy cost of the computation required to compress data can sometimes exceed the energy saved by reducing radio transmission. Moving toward energy-aware, asynchronous, and standardized communication protocols (such as MPEG-7 Part 17) represents the next frontier for the industry.

Summary updated

Abstract

Due to their great performance and scalability properties neural networks have become ubiquitous building blocks of many applications. With the rise of mobile and IoT, these models now are also being increasingly applied in distributed settings, where the owners of the data are separated by limited communication channels and privacy constraints. To address the challenges of these distributed environments, a wide range of training and evaluation schemes have been developed, which require the communication of neural network parametrizations. These novel approaches, which bring the "intelligence to the data" have many advantages over traditional cloud solutions such as privacy-preservation, increased security and device autonomy, communication efficiency and high training speed. This paper gives an overview over the recent advancements and challenges in this new field of research at the intersection of machine learning and communications.
BibTeX
@misc{sattler2020trends,
  author = {Felix Sattler and Thomas Wiegand and Wojciech Samek},
  title = {{Trends and Advancements in Deep Neural Network Communication}},
  year = {2020},
  eprint = {2003.03320},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.03320},
}
Figure