Abstract
Mobile edge learning is an emerging technique that enables distributed edge devices to collaborate in training shared machine learning models by exploiting their local data samples and communication and computation resources. To deal with the straggler dilemma issue faced in this technique, this paper proposes a new device-to-device (D2D) enabled data sharing approach, in which different edge devices share their data samples among each other over communication links, in order to properly adjust their computation loads for increasing the training speed. Under this setup, we optimize the radio resource allocation for both data sharing and distributed training, with the objective of minimizing the total training delay under fixed numbers of local and global iterations. Numerical results show that the proposed data sharing design significantly reduces the training delay, and also enhances the training accuracy when the data samples are non-independent and identically distributed (non-IID) among edge devices.
Introduction
Distributed Machine Learning (DML) at the wireless edge—often referred to as mobile edge learning—allows devices like smartphones and IoT sensors to train shared models without sending raw data to a central cloud. However, two major hurdles remain:
- The Straggler's Dilemma: Edge devices are heterogeneous. A single device with a slow processor or a poor wireless connection becomes a bottleneck, forcing the entire network to wait for it to finish its local update.
- Non-IID Data: In the real world, data is rarely "Independent and Identically Distributed." This lack of data diversity at the local level makes it difficult for the global model to converge and hurts final accuracy.
While previous research focused on compressing gradients or adjusting iteration counts, this paper tackles the problem at the source: the data distribution itself.
System Model and Methods
The authors introduce a D2D-enabled data sharing phase that occurs before the standard distributed training process.
The Workflow:
- Data Sharing: Devices exchange raw data samples over orthogonal frequency bands. Faster devices or those with better connections "take on" more data from slower devices to balance the future computation load.
- Distributed Training: The system uses Distributed Batch Gradient Descent (BGD). This involves four steps per global iteration:
- Global Broadcasting: The server sends the current model parameters to all devices.
- Local Update: Devices compute gradients based on their (now redistributed) local datasets.
- Local Uploading: Devices send their updates back to the server via Frequency Division Multiple Access (FDMA).
- Global Aggregation: The server averages the updates to improve the model.
The core contribution is a mathematical framework to minimize total training delay. The authors formulate a non-convex optimization problem that balances the time spent sharing data via D2D against the time saved during training. They successfully transform this into a convex form, allowing them to find the optimal radio resource allocation (bandwidth and power) and data transfer quantities.
Experimental Setup
The researchers evaluated their design using the MNIST dataset, a standard benchmark of 70,000 images of handwritten digits.
- Setup: 6 edge devices, each initially assigned 5,000 samples.
- Non-IID Condition: To simulate real-world imbalances, each device was restricted to samples from only two different digit classes.
- Model: A Convolutional Neural Network (CNN) with 7 layers.
Key Results
The proposed D2D-enabled design was compared against conventional designs using fixed and adaptive resource allocation without sharing.
- Reduced Latency: As shown in the figure above, in a scenario with 100 global iterations, the proposed design reduced total training delay from approximately 2200 seconds to 1500 seconds compared to fixed allocation.
- Superior Accuracy: Under non-IID conditions, the proposed design achieved significantly higher training accuracy compared to benchmark schemes within the same number of iterations, reaching roughly 90% accuracy.
- Scaling: The performance gains became more substantial as the number of global iterations ($M$) increased, showing that the initial "investment" in D2D sharing time pays off over the long run.
Conclusion and Future Work
D2D sharing effectively balances computation loads by shifting data from "stragglers" to "leaders," ensuring all devices finish their tasks at roughly the same time. Beyond speed, sharing data "mixes" labels across devices, mitigating the accuracy penalty typically seen in Federated Learning when data is non-IID.
Future work could address privacy concerns regarding raw data sharing, multi-objective optimization for energy efficiency, and the impact of device mobility on static channel gains.
Summary updated