TL;DR
This paper introduces Federated Generalized Category Discovery (Fed-GCD), a new decentralized learning task where privacy-protected clients collaboratively discover novel visual categories from partially labeled and unlabeled data. To overcome severe data heterogeneity across clients, the authors propose Associated Gaussian Contrastive Learning (AGCL), a framework that uses learnable Gaussian Mixture Models (GMMs) to safely aggregate global category knowledge on a central server without sharing raw data.
What Problem Does This Paper Address?
Generalized Category Discovery (GCD) is the task of clustering unlabeled data into both known and unknown categories, leveraging a limited set of labeled data from known classes. While highly useful, standard GCD algorithms assume a centralized architecture where all data is pooled together.
In reality, data is often distributed across different organizations or devices (clients) that cannot share raw data due to strict privacy laws or proprietary constraints. For example, different wildlife research stations globally might want to collaborate to discover new bird species without violating local data-sharing regulations.
This paper introduces Federated GCD (Fed-GCD) to address this gap. Fed-GCD introduces two major bottlenecks:
- Representation Degradation: Individual clients train on much smaller, localized datasets, harming the model's ability to learn robust features.
- Label Space Heterogeneity: The categories present on each client can be highly inconsistent, meaning some clients might hold data for completely non-overlapping sets of classes.
Methods
To tackle these challenges, the authors present the Associated Gaussian Contrastive Learning (AGCL) framework, which moves away from standard instance-level or single-prototype contrastive learning. Instead, it models categories using learnable Gaussian Mixture Models (GMMs).
The framework operates via three core components:
- Semi-FINCH for Local Initialization: Clients use an enhanced, semi-supervised version of the parameter-free FINCH clustering algorithm to assign pseudo-labels to local unlabeled data. This allows them to estimate the initial mean and covariance for their local GMMs.
- Client Semantics Association (CSA): Local GMM parameters are uploaded to a central server. The server samples feature-level instances from these distributions and clusters them to form a comprehensive global GMM. This process implicitly aligns shared classes across clients and maps out broader inter-class relationships.
- Global-Local GMM Contrastive Learning (GCL): The global GMM is sent back to the clients. Clients then optimize their local models using a combined loss function that contrasts instances against both local and global GMM distributions. A Mahalanobis distance metric is used to dynamically adjust contrastive temperatures based on cluster variances, making the model resilient to noisy pseudo-labels.
Datasets
The authors established a new Fed-GCD benchmark by reorganizing six widely used computer vision datasets:
- Generic Datasets: CIFAR-10, CIFAR-100, and ImageNet-100.
- Fine-Grained Datasets: CUB-200 (birds), Stanford Cars, and Oxford-IIIT Pet.
To simulate real-world conditions, they used a $\beta$-Dirichlet distribution to partition the data into two distinct evaluation protocols: Normally Heterogeneous (NH, $\beta=0.2$), where a few common classes are shared, and Extremely Heterogeneous (EH, $\beta=0.05$), where clients share virtually no labeled categories.
Key Insights and Ideas
- GMMs as a Privacy-Preserving Generative Bridge: Instead of transferring raw images or complex generative models, transferring simple GMM parameters (means and covariances) allows the server to synthesize informative feature-level samples. This achieves knowledge aggregation without privacy leaks.
- Beyond Single-Vector Prototypes: Traditional prototypical contrastive learning represents a class as a single average vector. By using GMMs, AGCL captures class-specific variance. This allows the model to utilize the Mahalanobis distance, effectively down-weighting unreliable, scattered clusters and preventing out-of-distribution outliers from skewing the representation space.
- Complementary Global-Local Supervision: Local GCL ensures the model remains highly accurate on the client's specific data distribution, while global GCL introduces category relationships that a single client might never observe locally.
Key Results
The proposed AGCL framework consistently outperformed the standard FedAvg baseline across all datasets and heterogeneity settings:
- Generic Datasets: In the NH setting on CIFAR-100, AGCL achieved a 56.1% accuracy for all classes, outperforming the FedAvg+GCD baseline by 6.5%. Even in the brutal EH setting (no shared labeled classes), AGCL improved performance by 6.9% on CIFAR-100 and 5.7% on ImageNet-100.
- Fine-Grained Datasets: Fine-grained tasks showed the most significant gains. On the CUB-200 dataset, AGCL outperformed the baseline by 8.9% in the NH setting and 9.8% in the EH setting, proving highly effective at distinguishing subtle differences between sub-categories in a decentralized manner.
Limitations and Future Work
The authors note that the current setup still relies on storing a small set of labeled data on the central server to estimate the total number of potential categories ($k$) during testing. A key direction for future work is to completely relieve this requirement, allowing for a fully decentralized category estimation process that fits even more realistic deployment scenarios.
Summary updated