TL;DR
This paper introduces FedDCT, a novel federated learning approach that enables resource-constrained edge devices to train large deep convolutional neural networks (CNNs). By dividing a large model into smaller sub-models and training them collaboratively within clusters of devices, FedDCT significantly reduces client memory requirements and communication overhead, achieving state-of-the-art performance on various datasets.
What Problem Does This Paper Address?
Federated Learning (FL) allows devices to collaboratively train models without sharing raw data, preserving privacy. However, a major hurdle is the increasing size of state-of-the-art CNNs, which demand significant memory and computational resources. This makes it difficult for resource-limited edge devices like smartphones to participate effectively in training large models. Existing FL methods often either use smaller, less accurate models or assume clients have ample resources, leaving a gap for training large, high-performance models on constrained devices.
Methods
The core innovation of FedDCT is its "Divide and Co-Training" strategy. Instead of each client training the entire large model, FedDCT:
- Model Division: Splits a large global CNN into an ensemble of smaller, equally-sized sub-models. This division is done channel-wise to maintain the overall parameter and FLOP count.
- Collaborative Training within Clusters: Groups clients into clusters. Within each cluster, clients collaboratively train the ensemble of sub-models. One client acts as the "main device" for generating augmented data and performing forward propagation up to a "cut layer." The intermediate activations (smashed data) are then shared among other "proxy" clients in the cluster, who complete the forward propagation and backpropagation on their respective sub-models. This process is designed to maintain data privacy as raw data is never shared.
- Mutual Learning: Clients within a cluster learn from each other's predictions through a co-training loss (Jensen-Shannon divergence), further improving ensemble performance.
- Federated Aggregation: After cluster-level training, the server aggregates the updated sub-models from all clusters to form a new global ensemble model.
This approach significantly reduces the memory footprint on individual clients, as they only need to handle a fraction of the full model.
Datasets
The paper evaluates FedDCT on the following datasets:
- CIFAR-10: A standard benchmark for image classification.
- CIFAR-100: Similar to CIFAR-10 but with a larger number of classes.
- HAM10000: A real-world dataset of dermatoscopic images for skin lesion classification, noted for its class imbalance.
- VAIPE: A newly collected and annotated dataset for pill image identification, designed to test robustness in real-world settings.
Key Insights and Ideas
- Decoupling Model Size from Client Resources: FedDCT effectively decouples the size of the global model from the memory constraints of individual clients by distributing the model's components.
- Collaborative Learning Enhances Performance: The co-training mechanism within clusters, where sub-models learn from each other's predictions on different data views, leads to improved ensemble performance beyond what individual sub-models could achieve.
- Privacy-Preserving Split Learning: FedDCT integrates principles of split learning within a federated setting, ensuring that raw data remains on the client and only intermediate activations are shared within a cluster, thus enhancing privacy.
- Reduced Communication Overhead: By training smaller sub-models and aggregating them, FedDCT significantly reduces the number of communication rounds needed to reach target accuracy compared to traditional FL methods.
Key Results
- State-of-the-Art Accuracy: FedDCT consistently achieves state-of-the-art Top-1 accuracy across all tested datasets (CIFAR-10, CIFAR-100, HAM10000, VAIPE), outperforming methods like FedAvg, FedProx, SplitFed, and FedGKT. For example, on CIFAR-10, it reached 95.99% accuracy with WRN-16-8.
- Reduced Memory Requirements: FedDCT offers substantial memory savings, with clients requiring significantly less memory (e.g., less than 1GB for S=16 split factor on CIFAR-100 with ResNet-110) compared to FedAvg (around 4.4GB).
- Faster Convergence: The method requires substantially fewer communication rounds to reach target accuracies. For instance, on the VAIPE dataset, FedDCT reached 80% accuracy in 67 rounds, compared to 195 rounds for FedAvg.
- No Server-Side Training Cost: Unlike some other distributed methods, FedDCT does not impose additional computational burden on the central server for training.
Limitations and Future Work
- CNN Focus: The current FedDCT framework is primarily designed for CNNs. Extending it to other architectures like Recurrent Neural Networks (RNNs) or U-Nets is a potential future direction.
- Slowest Client Bottleneck: Similar to other FL methods, the slowest client in a cluster dictates the pace of training. The authors suggest incorporating advanced client selection methods to mitigate this.
- Heterogeneous Resources: While FedDCT addresses memory constraints, it assumes clients can complete their assigned sub-model training. Future work could explore dividing models into unequal partitions based on device capabilities.
- Advanced Collaborative Training: The paper mentions that more sophisticated collaborative training techniques could further enhance performance.
- Privacy Enhancements: While FedDCT offers inherent privacy benefits, the authors suggest integrating differential privacy (DP) or multi-party computation (MPC) for stronger protection against advanced adversaries.
Summary updated