FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training

Quan Nguyen, Hieu H. Pham, Kok-Seng Wong, Phi Le Nguyen, Truong Thao Nguyen, Minh N. Do

TL;DR

FedDCT enables resource-constrained edge devices (like smartphones and wearables) to participate in training large-scale convolutional neural networks (CNNs) by dividing a single massive model into an ensemble of smaller sub-models. By grouping clients into collaborative clusters, the framework reduces local memory requirements by 4–8x while achieving higher accuracy and faster convergence than current state-of-the-art federated learning methods.

What Problem Does This Paper Address?

Federated Learning (FL) is designed to train models across decentralized devices while keeping data private. However, it hits a "memory wall" in practice: modern, high-performance CNNs are incredibly parameter-heavy. Standard FL requires every participating device to train the entire model locally.

As shown in Figure 1, training a state-of-the-art model like SeNet-154 can require nearly 20GB of RAM—far exceeding the 2GB to 6GB typically found in the majority of mobile devices. Existing workarounds either use "tiny" models with poor accuracy or rely on "Split Learning," which offloads heavy computation to a central server, creating a massive bottleneck when scaling to millions of devices.

Figure 1

Methods

The authors introduce FedDCT (Federated Divide and Collaborative Training), which reimagines the global model not as one giant network, but as an ensemble of $S$ smaller, parallel sub-models.

  1. Model Division: A large CNN is divided channel-wise into $S$ equal-sized sub-networks. For example, a ResNet bottleneck layer can be split into four smaller blocks, each with a fraction of the original parameters and FLOPs.
  2. Clustered Collaborative Training: Clients are grouped into clusters of size $S$. Instead of training the whole model, each client in a cluster trains only one sub-model.
  3. Privacy-Preserving Split: To maintain privacy without sharing raw data between clients in a cluster, FedDCT uses a "cut layer" approach. The "main client" (who holds the data) runs the first few layers, sends the resulting activations ("smashed data") to "proxy clients" in the cluster, who then complete the forward pass on their respective sub-models. Gradients are passed back similarly during backpropagation. This process is illustrated in Figure 2 and Figure 4.
  4. Mutual Knowledge Distillation: To ensure these sub-models work well together, the authors employ different data augmentations for each sub-network and add a Jensen-Shannon (JS) divergence loss. This encourages sub-models to learn from each other’s predictions, improving the overall ensemble performance.
  5. Server Aggregation: The central server simply performs a weighted average of the ensemble models received from all clusters, incurring zero extra training cost compared to standard FedAvg.
Fig. 2: Illustration of a typical FL architecture (left) and the proposed FedDCT (right). FedDCT enables a cluster of clients to cooperatively train a big deep learning model by breaking it into an ensemble of several small sub-models and training these sub-models on several devices concurrently while maintaining client data privacy. We refer to this algorithm as Federated Divide and Collaborative Training or FedDCT in short.
Figure 4

Datasets

The framework was evaluated on four image classification datasets:

  • CIFAR-10 & CIFAR-100: Standard benchmarks for natural image classification.
  • HAM10000: A large-scale medical dataset of multi-source dermatoscopic skin lesion images.
  • VAIPE: A new, real-world pill image identification dataset introduced by the authors, containing 98 pill categories photographed under unconstrained environments.

Key Insights and Ideas

  • Ensembles as a Feature, Not a Bug: Rather than just splitting a model to save memory, FedDCT leverages the fact that ensembles of diverse sub-models often outperform a single large model of equivalent size.
  • Solving the Server Bottleneck: Unlike SplitFed or Split Learning, which require the server to handle the "heavy lifting" of training for every client, FedDCT keeps the training workload distributed among the edge devices. The server's role remains simple aggregation.
  • Communication Efficiency: While there is a slight overhead from exchanging activations within a cluster, FedDCT converges so much faster (in terms of communication rounds) that the total communication cost is often lower than standard FL.
  • Privacy through Architecture: By combining model splitting with clustered training, FedDCT ensures that raw data never leaves the originating device, and proxy clients only ever see intermediate activations.

Key Results

  • Memory Reduction: FedDCT achieved a 4–8x reduction in client memory requirements. For a ResNet-110 model on CIFAR-100, memory usage dropped from ~4.4GB (standard FL) to less than 1GB (FedDCT with $S=16$).
  • Superior Accuracy: FedDCT consistently outperformed SOTA methods like FedAvg, FedProx, and SplitFed. On the VAIPE dataset, FedDCT achieved 97.75% accuracy, significantly surpassing SplitFed (91.85%) and FedAvg (88.79%).
  • Faster Convergence: On the VAIPE dataset, FedDCT reached 80% accuracy in just 67 rounds, compared to 195 rounds for FedAvg—a 3x speedup.
  • Server Efficiency: As shown in the paper's efficiency evaluation, FedDCT requires zero training overhead on the server side, whereas competing split-learning methods require massive server-side VRAM (often >400GB for 100 clients).

Limitations and Future Work

  • Architecture Constraints: The current division strategy is optimized for CNNs and is difficult to apply to time-series models (RNNs) or certain segmentation networks (like U-Nets).
  • Synchronous Bottleneck: Within a cluster, the training pace is dictated by the slowest device ("straggler" problem).
  • Heterogeneous Resources: The current model assumes equal splits. Future work could explore unequal partitions, assigning larger sub-models to more powerful devices within a cluster.
  • Advanced Privacy: While inherently more private than standard FL, the authors suggest integrating Differential Privacy (DP) or Multi-Party Computation (MPC) to further protect against sophisticated gradient-inversion attacks.

Summary updated

Abstract

We introduce FedDCT, a novel distributed learning paradigm that enables the usage of large, high-performance CNNs on resource-limited edge devices. As opposed to traditional FL approaches, which require each client to train the full-size neural network independently during each training round, the proposed FedDCT allows a cluster of several clients to collaboratively train a large deep learning model by dividing it into an ensemble of several small sub-models and train them on multiple devices in parallel while maintaining privacy. In this collaborative training process, clients from the same cluster can also learn from each other, further improving their ensemble performance. In the aggregation stage, the server takes a weighted average of all the ensemble models trained by all the clusters. FedDCT reduces the memory requirements and allows low-end devices to participate in FL. We empirically conduct extensive experiments on standardized datasets, including CIFAR-10, CIFAR-100, and two real-world medical datasets HAM10000 and VAIPE. Experimental results show that FedDCT outperforms a set of current SOTA FL methods with interesting convergence behaviors. Furthermore, compared to other existing approaches, FedDCT achieves higher accuracy and substantially reduces the number of communication rounds (with $4-8$ times fewer memory requirements) to achieve the desired accuracy on the testing dataset without incurring any extra training cost on the server side.
BibTeX
@article{nguyen2024feddct,
  author = {Quan Nguyen and Hieu H. Pham and Kok-Seng Wong and Phi Le Nguyen and Truong Thao Nguyen and Minh N. Do},
  title = {{FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training}},
  journal = {IEEE Trans. Netw. Serv. Manag.},
  year = {2024},
  doi = {10.1109/tnsm.2023.3314066},
}
Figure