Abstract: Federated Generalized Category Discovery (Fed-GCD) is a novel task that requires decentralized clients to categorize both known and unknown classes without sharing raw data. This paper introduces Associated Gaussian Contrastive Learning (AGCL), a framework utilizing learnable Gaussian Mixture Models (GMMs) to bridge client gaps. By aggregating class-level variance and mean through GMMs, AGCL enables robust feature learning and global category alignment. Experiments on generic and fine-grained datasets demonstrate that AGCL significantly outperforms standard federated baselines.
What Problem Does This Paper Address?
In computer vision, Generalized Category Discovery (GCD) is often considered a "holy grail" of classification. The goal is to take a set of unlabeled samples and group them into both known classes (from a small labeled set) and entirely new, unknown categories.
Traditionally, GCD research has assumed a centralized environment where all data is accessible on a single server. However, real-world data—such as medical records or ecological observations—is frequently distributed across different institutions (clients) that cannot share raw data due to privacy regulations. This decentralized reality introduces two major hurdles:
- Representation Degradation: Individual clients often possess limited data, making it difficult for models to learn robust, generalizable features.
- Label Heterogeneity: Different clients observe different sets of categories based on their location or specialization (e.g., a bird-watching station in Asia sees different species than one in Europe). Building a unified global model under these conditions is exceptionally difficult.
Methods: Associated Gaussian Contrastive Learning (AGCL)
To address these challenges, the authors propose Associated Gaussian Contrastive Learning (AGCL). Unlike standard federated learning (like FedAvg) which simply averages model weights, AGCL focuses on capturing and aligning class-level relationships through a more sophisticated statistical lens.
The AGCL framework consists of several key components designed to work in concert across the client-server boundary:
- Learnable GMMs: Rather than representing a class as a single point (a prototype), AGCL models each class as a Gaussian Mixture Model (GMM). This approach accounts for the variance and "spread" of a class in the feature space. By using Mahalanobis distance, the model performs more robust, class-aware contrastive learning that is less sensitive to outliers.
- Client Semantics Association (CSA): The central server aggregates the GMM parameters (means and variances) uploaded by clients. By sampling feature-level instances from these GMMs and re-clustering them, the server constructs a "Global GMM." This aligns shared classes across different clients and builds a comprehensive knowledge base without the server ever seeing a raw image.
- Global-Local GMM Contrastive Learning (GCL): Once the global GMM is sent back to the clients, the local models perform contrastive learning using both their local GMM and the server’s global view. This forces local models to respect global category relationships, effectively providing "supervision" for classes the client might not have in its own dataset.
- Semi-FINCH: To initialize these GMMs without a predefined number of classes, the authors developed a semi-supervised version of the FINCH clustering algorithm, allowing for flexible discovery of unknown categories.
Datasets and Experimental Setup
The authors established a new benchmark for Fed-GCD by reorganizing six popular datasets into federated partitions:
- Generic Datasets: CIFAR-10, CIFAR-100, and ImageNet-100.
- Fine-Grained Datasets: CUB-200 (birds), Stanford Cars, and Oxford-IIIT Pet.
To simulate realistic data distributions, they employed a Dirichlet distribution (controlled by parameter $\beta$) to split the data. This created scenarios ranging from "normally heterogeneous" (some shared classes) to "extremely heterogeneous" (almost no shared classes between clients), testing the model's ability to generalize in the toughest conditions.
Key Insights and Results
The AGCL framework consistently outperformed the standard FedAvg + GCD baseline across all benchmarks:
- Variance Matters: Traditional prototypical learning treats every class as a uniform sphere. By using GMMs, AGCL acknowledges that some classes are more diverse than others, making the model significantly more robust to noisy or incorrectly clustered samples.
- Feature-Level Collaboration: The research demonstrates that raw images are not necessary for effective collaboration. Sharing GMM parameters allows clients to "describe" their classes to the server, which then synthesizes a global perspective that benefits all participants.
- Performance Gains: On CIFAR-100, AGCL improved accuracy by 6.5% to 6.9%. The gains were even more significant in fine-grained tasks like the CUB-200 bird dataset, where AGCL outperformed the baseline by 8.9% to 9.8%.
Limitations and Future Work
While AGCL represents a major advancement, the authors identified several areas for future research. Currently, the method relies on a ViT backbone pre-trained on ImageNet; future work could explore training such models from scratch in a federated environment. Additionally, the current setup assumes the server has access to some labeled data for "Old" classes to facilitate testing. Moving toward a scenario where the server requires even less labeled data remains a goal. Finally, the communication overhead of frequent GMM updates in massive-scale deployments is an open question for optimization.
Summary updated