Cooperative Learning VIA Federated Distillation OVER Fading Channels

Jin-Hyun Ahn, Osvaldo Simeone, Joonhyuk KangICASSP 2020 · DOI

TL;DR

This paper proposes and analyzes new wireless communication protocols for Federated Distillation (FD), a more communication-efficient approach to distributed machine learning than standard Federated Learning (FL). They show that FD and its enhanced version, Hybrid FD (HFD), significantly outperform FL in scenarios with limited wireless resources, especially when using analog communication techniques.

What Problem Does This Paper Address?

Distributed machine learning, particularly Federated Learning (FL), allows multiple devices to collaboratively train a model without sharing their raw data. However, FL requires frequent communication of model parameters (weights) between devices and a central server, which can be a bottleneck, especially over wireless channels. Federated Distillation (FD) was proposed as a more communication-efficient alternative, exchanging only model outputs (logits) instead of weights. This paper tackles the challenge of implementing FD and an enhanced version (HFD) over realistic wireless fading channels, considering both digital and analog communication strategies to optimize spectral efficiency.

Methods

The authors propose four distinct wireless communication protocols for FL, FD, and HFD:

  • Digital-Digital (D-D): Both uplink and downlink use standard digital communication with separate source-channel coding.
  • Digital-Analog (D-A): Uplink is digital, downlink is analog (over-the-air computing).
  • Analog-Digital (A-D): Uplink is analog, downlink is digital.
  • Analog-Analog (A-A): Both uplink and downlink use analog communication.

For digital transmissions, they employ techniques like sparse binary compression with error accumulation. For analog transmissions, they leverage over-the-air computing, where devices transmit signals simultaneously, and the central server decodes the sum directly. They also consider dimensionality reduction techniques when the number of transmitted samples exceeds the available channel uses.

Datasets

The paper evaluates their proposed methods using the MNIST dataset for image classification. They split the training data into disjoint sets for each of the 10 devices, resulting in unbalanced datasets with respect to the ten classes. For evaluation, they use 10,000 randomly selected images from the MNIST dataset.

Key Insights and Ideas

  • FD and HFD offer significant advantages over FL in limited spectral resources: When the number of available channel uses (T) is low, FD and HFD achieve much higher test accuracies than FL, highlighting their communication efficiency.
  • Analog communication (A-A) is particularly beneficial for FL: For standard Federated Learning, the analog-analog (A-A) protocol consistently outperforms other digital and hybrid schemes, especially at low spectral resource regimes.
  • HFD uniformly improves upon FD: The enhanced Hybrid FD (HFD) consistently shows better performance than FD across various communication scenarios.
  • Over-the-air computing is a powerful tool for wireless ML: The use of analog communication for over-the-air computing demonstrates its potential to improve the efficiency of distributed learning over wireless channels.

Key Results

  • In Figure 2, which plots test accuracy against the number of channel uses (T), FD and HFD significantly outperform FL at low values of T. For instance, at T=100, HFD achieves around 0.7 accuracy, while FL is below 0.5. The A-A scheme is shown to be the best for FL implementations.
  • Figure 3, which plots test accuracy against SNR, confirms that FD and HFD significantly outperform FL at low SNR values. The A-A scheme again shows the best performance, particularly for lower SNR values.
  • HFD consistently provides a performance improvement over FD across the tested scenarios.

Limitations and Future Work

  • Channel State Information (CSI): The paper assumes knowledge of channel state information (CSI) for both the uplink and downlink, which can be challenging to obtain in real-world wireless environments.
  • Power Allocation: While full-power transmission is considered for simplicity in analog communication, the authors mention that extensions to optimized power control are conceptually straightforward.
  • Unbalanced Datasets: The experiments use unbalanced datasets for each device, which is a realistic scenario but might not fully capture the performance in perfectly balanced settings.

Future work could explore scenarios with imperfect CSI, investigate different power allocation strategies for analog communication, and extend the analysis to more complex network topologies and machine learning tasks.

Summary updated

BibTeX
@inproceedings{ahn2020cooperative,
  author = {Jin-Hyun Ahn and Osvaldo Simeone and Joonhyuk Kang},
  title = {{Cooperative Learning VIA Federated Distillation OVER Fading Channels}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053448},
}
Figure