Research on Asynchronous Distributed Deep Learning Technology―Optimizing Machine Learning Models in the Age of Distributed Data Storage

Kenta NiwaNTT 2021 · DOI

Abstract

As deep learning scales, centralizing massive datasets becomes impractical due to data volume and privacy regulations like GDPR. This paper introduces "Edge-consensus Learning," a novel asynchronous distributed algorithm that enables collaborative model training across decentralized servers. Unlike conventional methods that struggle with statistically biased (heterogeneous) data, the proposed approach allows nodes to synthesize learning elements asynchronously, preserving privacy while achieving performance near that of centralized models.

Introduction: The Decentralization Imperative

Modern deep learning typically relies on aggregating massive amounts of data into a single, centralized datacenter. However, this paradigm faces two major roadblocks: the sheer volume of data generated by edge devices (e.g., self-driving cars, smart grids) and strict privacy regulations (e.g., GDPR) that prohibit the sharing of sensitive information like medical records. Distributed deep learning offers a solution by training models where the data resides, but traditional methods like Gossip-SGD rely on synchronous communication and assume data is statistically similar across nodes—assumptions that rarely hold in real-world "heterogeneous" environments.

Proposed Method: Edge-consensus Learning

To address these challenges, we propose "Edge-consensus Learning" (ECL). Unlike simple model averaging, which can lead to divergence when nodes have conflicting data distributions, ECL mathematically models multi-node collaboration. It allows servers to communicate asynchronously, synthesizing individual learning elements without requiring nodes to wait for global synchronization. This approach accommodates the statistical biases of local data, effectively allowing a network of "specialists" to collaborate into a "generalist" model.

Simulation of decentralized training across eight global nodes for medical imaging diagnosis.

Experimental Evaluation

The proposed method was evaluated using two benchmarks:

  1. CIFAR-10: A standard image classification task distributed across 8 servers with intentional statistical bias to simulate real-world heterogeneity.
  2. Medical Imaging: A diagnostic aid model for 14 diseases, simulated across a network of 8 hospitals (N1 to N8) with regional disease distribution biases.

Key Results

On the CIFAR-10 benchmark, Edge-consensus Learning significantly outperformed the conventional Gossip-SGD method. While Gossip-SGD struggled to converge on heterogeneous data, ECL's performance closely approached the "global reference model"—a theoretical upper bound representing a model trained on centrally aggregated data.

Comparison of error rates between Edge-consensus Learning, Gossip-SGD, and a global reference model.

In the medical imaging demonstration, the algorithm successfully trained a high-performance model across the simulated hospital network. Despite each hospital seeing different disease frequencies, the decentralized model achieved high recognition accuracy across all nodes, proving its viability for privacy-sensitive applications.

Conclusion and Future Outlook

Edge-consensus Learning provides a scalable, privacy-preserving framework for the future of decentralized computing. Future work involves deploying this technology across physical communication stations to create a massive edge computing network. This infrastructure will support the IOWN (Innovative Optical and Wireless Network) concept, enabling real-time, personalized control for applications like autonomous vehicle coordination in smart cities.

Summary updated

Abstract

Overview
BibTeX
@inproceedings{niwa2021research,
  author = {Kenta Niwa},
  title = {{Research on Asynchronous Distributed Deep Learning Technology―Optimizing Machine Learning Models in the Age of Distributed Data Storage}},
  booktitle = {NTT},
  year = {2021},
  doi = {10.53829/ntr202109ri1},
}
Figure