Abstract
Federated learning aims to tackle the "isolated data island" problem, where it trains a collective model from physically isolated clients while safeguarding the privacy of users' data. However, supervised federated learning necessitates that each client labels their data for training, which can be both time-consuming and resource-intensive, and may even be impractical for edge devices. Moreover, the training and transmission of deep models present challenges to the computation and communication capabilities of the clients. To address these two inherent challenges in supervised federated learning, we propose a novel lightweight unsupervised federated learning approach that leverages unlabeled data on each client to perform lightweight model training and communication by harnessing pretrained vision-language models, such as CLIP. By capitalizing on the zero-shot prediction capability and the well-trained image encoder of the pre-trained CLIP model, we have carefully crafted an efficient and resilient self-training approach. This method refines the initial zero-shot predicted pseudo-labels of unlabeled instances through the sole training of a linear classifier on top of the fixed image encoder. Additionally, to address data heterogeneity within each client, we propose a class-balanced text feature sampling strategy for generating synthetic instances in the feature space to support local training. Experiments are conducted on multiple benchmark datasets. The experimental results demonstrate that our proposed method greatly enhances model performance in comparison to CLIP's zero-shot predictions and even outperforms supervised federated learning benchmark methods given limited computational and communication overhead.
Introduction
Standard Federated Learning (FL) faces three major hurdles in real-world deployments: the labeling burden, resource constraints on edge devices, and data heterogeneity (Non-IID). Most FL frameworks assume supervised learning, which is often impractical for end-users. Furthermore, training large models on edge devices consumes significant bandwidth and compute. We propose Federated Self-Training with Class-Balanced Data Generation (FST-CBDG), a lightweight framework for unsupervised federated learning that eliminates the need for data labeling on client devices by leveraging pretrained vision-language models (VLMs) like CLIP.
Methodology
The FST-CBDG framework follows a three-stage pipeline designed for efficiency and resilience:
- Server-Side Preparation: The server utilizes a pretrained CLIP text encoder to generate embeddings for all potential class names (e.g., "a photo of a dog"). These embeddings serve as initial weights for a lightweight linear classifier distributed to clients.
- Lightweight Local Training: Clients keep the CLIP image encoder fixed, training only the linear classifier. To handle unlabeled data, a self-training approach is employed where the model generates its own pseudo-labels. These labels are refined using a weighted moving average ($q_j^t = \beta q_j^{t-1} + (1-\beta)f(I_j)$) to ensure training stability.
- Class-Balanced Data Generation (CBDG): To address data heterogeneity, CLIP’s text embeddings are treated as "class prototypes." Clients sample synthetic features from a Gaussian distribution centered around these prototypes, allowing the model to learn about missing classes in the local feature space.
Experimental Setup
The approach was evaluated on three computer vision benchmarks: CIFAR-10, CIFAR-100, and CINIC-10, partitioned among 100 clients. Data was split using both homogeneous and heterogeneous strategies (Sharding and Latent Dirichlet Allocation) to simulate real-world imbalances.
Key Results and Insights
FST-CBDG demonstrates significant performance gains over both zero-shot baselines and supervised FL methods:
- Performance Gains: On CIFAR-10, FST-CBDG improved CLIP's zero-shot accuracy from 68.7% to 74.0%.
- Competitive with Supervised FL: In lightweight settings, FST-CBDG outperformed standard supervised methods like FedAvg and FedNTD. For example, on CIFAR-100 with heterogeneous data, FST-CBDG reached 43.3% accuracy, while FedAvg achieved 34.1%.
- Efficiency: The model often reaches near-optimal performance in just 1 to 6 communication rounds, drastically reducing energy and bandwidth requirements.
Conclusion
FST-CBDG offers a robust, unsupervised solution for federated learning by capitalizing on the alignment of text and image features in VLMs. While highly effective, the method's success depends on the quality of the pretrained VLM and requires knowledge of class names. Future research will explore lightweight adaptation of the image encoder to further push performance boundaries.
Summary updated