FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Co-Training

Quan Nguyen, Hieu H. Pham, Kok-Seng Wong, Phi Le Nguyen, Truong Thao Nguyen, Minh N. Do

TL;DR

This paper introduces FedDCT, a novel federated learning approach that enables resource-constrained edge devices to train large deep convolutional neural networks (CNNs). By dividing a large model into smaller sub-models and training them collaboratively within clusters of devices, FedDCT significantly reduces client memory requirements and communication overhead, achieving state-of-the-art performance on various datasets.

What Problem Does This Paper Address?

Federated Learning (FL) allows devices to collaboratively train models without sharing raw data, preserving privacy. However, a major hurdle is the increasing size of state-of-the-art CNNs, which demand significant memory and computational resources. This makes it difficult for resource-limited edge devices like smartphones to participate effectively in training large models. Existing FL methods often either use smaller, less accurate models or assume clients have ample resources, leaving a gap for training large, high-performance models on constrained devices.

Methods

The core innovation of FedDCT is its "Divide and Co-Training" strategy. Instead of each client training the entire large model, FedDCT:

  • Model Division: Splits a large global CNN into an ensemble of smaller, equally-sized sub-models. This division is done channel-wise to maintain the overall parameter and FLOP count.
  • Collaborative Training within Clusters: Groups clients into clusters. Within each cluster, clients collaboratively train the ensemble of sub-models. One client acts as the "main device" for generating augmented data and performing forward propagation up to a "cut layer." The intermediate activations (smashed data) are then shared among other "proxy" clients in the cluster, who complete the forward propagation and backpropagation on their respective sub-models. This process is designed to maintain data privacy as raw data is never shared.
  • Mutual Learning: Clients within a cluster learn from each other's predictions through a co-training loss (Jensen-Shannon divergence), further improving ensemble performance.
  • Federated Aggregation: After cluster-level training, the server aggregates the updated sub-models from all clusters to form a new global ensemble model.

This approach significantly reduces the memory footprint on individual clients, as they only need to handle a fraction of the full model.

Datasets

The paper evaluates FedDCT on the following datasets:

  • CIFAR-10: A standard benchmark for image classification.
  • CIFAR-100: Similar to CIFAR-10 but with a larger number of classes.
  • HAM10000: A real-world dataset of dermatoscopic images for skin lesion classification, noted for its class imbalance.
  • VAIPE: A newly collected and annotated dataset for pill image identification, designed to test robustness in real-world settings.

Key Insights and Ideas

  • Decoupling Model Size from Client Resources: FedDCT effectively decouples the size of the global model from the memory constraints of individual clients by distributing the model's components.
  • Collaborative Learning Enhances Performance: The co-training mechanism within clusters, where sub-models learn from each other's predictions on different data views, leads to improved ensemble performance beyond what individual sub-models could achieve.
  • Privacy-Preserving Split Learning: FedDCT integrates principles of split learning within a federated setting, ensuring that raw data remains on the client and only intermediate activations are shared within a cluster, thus enhancing privacy.
  • Reduced Communication Overhead: By training smaller sub-models and aggregating them, FedDCT significantly reduces the number of communication rounds needed to reach target accuracy compared to traditional FL methods.

Key Results

  • State-of-the-Art Accuracy: FedDCT consistently achieves state-of-the-art Top-1 accuracy across all tested datasets (CIFAR-10, CIFAR-100, HAM10000, VAIPE), outperforming methods like FedAvg, FedProx, SplitFed, and FedGKT. For example, on CIFAR-10, it reached 95.99% accuracy with WRN-16-8.
  • Reduced Memory Requirements: FedDCT offers substantial memory savings, with clients requiring significantly less memory (e.g., less than 1GB for S=16 split factor on CIFAR-100 with ResNet-110) compared to FedAvg (around 4.4GB).
  • Faster Convergence: The method requires substantially fewer communication rounds to reach target accuracies. For instance, on the VAIPE dataset, FedDCT reached 80% accuracy in 67 rounds, compared to 195 rounds for FedAvg.
  • No Server-Side Training Cost: Unlike some other distributed methods, FedDCT does not impose additional computational burden on the central server for training.

Limitations and Future Work

  • CNN Focus: The current FedDCT framework is primarily designed for CNNs. Extending it to other architectures like Recurrent Neural Networks (RNNs) or U-Nets is a potential future direction.
  • Slowest Client Bottleneck: Similar to other FL methods, the slowest client in a cluster dictates the pace of training. The authors suggest incorporating advanced client selection methods to mitigate this.
  • Heterogeneous Resources: While FedDCT addresses memory constraints, it assumes clients can complete their assigned sub-model training. Future work could explore dividing models into unequal partitions based on device capabilities.
  • Advanced Collaborative Training: The paper mentions that more sophisticated collaborative training techniques could further enhance performance.
  • Privacy Enhancements: While FedDCT offers inherent privacy benefits, the authors suggest integrating differential privacy (DP) or multi-party computation (MPC) for stronger protection against advanced adversaries.

Summary updated

Abstract

We introduce FedDCT, a novel distributed learning paradigm that enables the usage of large, high-performance CNNs on resource-limited edge devices. As opposed to traditional FL approaches, which require each client to train the full-size neural network independently during each training round, the proposed FedDCT allows a cluster of several clients to collaboratively train a large deep learning model by dividing it into an ensemble of several small sub-models and train them on multiple devices in parallel while maintaining privacy. In this co-training process, clients from the same cluster can also learn from each other, further improving their ensemble performance. In the aggregation stage, the server takes a weighted average of all the ensemble models trained by all the clusters. FedDCT reduces the memory requirements and allows low-end devices to participate in FL. We empirically conduct extensive experiments on standardized datasets, including CIFAR-10, CIFAR-100, and two real-world medical datasets HAM10000 and VAIPE. Experimental results show that FedDCT outperforms a set of current SOTA FL methods with interesting convergence behaviors. Furthermore, compared to other existing approaches, FedDCT achieves higher accuracy and substantially reduces the number of communication rounds (with $4-8$ times fewer memory requirements) to achieve the desired accuracy on the testing dataset without incurring any extra training cost on the server side.
BibTeX
@article{nguyen2024feddct,
  author = {Quan Nguyen and Hieu H. Pham and Kok-Seng Wong and Phi Le Nguyen and Truong Thao Nguyen and Minh N. Do},
  title = {{FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Co-Training}},
  journal = {IEEE Trans. Netw. Serv. Manag.},
  year = {2024},
  doi = {10.1109/tnsm.2023.3314066},
}
Figure