Federated Generalized Category Discovery

Nan Pu, Zhun Zhong, Xinyuan Ji, Nicu Sebe

TL;DR

This paper introduces Federated Generalized Category Discovery (Fed-GCD), a new decentralized learning task where privacy-protected clients collaboratively discover novel visual categories from partially labeled and unlabeled data. To overcome severe data heterogeneity across clients, the authors propose Associated Gaussian Contrastive Learning (AGCL), a framework that uses learnable Gaussian Mixture Models (GMMs) to safely aggregate global category knowledge on a central server without sharing raw data.

What Problem Does This Paper Address?

Generalized Category Discovery (GCD) is the task of clustering unlabeled data into both known and unknown categories, leveraging a limited set of labeled data from known classes. While highly useful, standard GCD algorithms assume a centralized architecture where all data is pooled together.

In reality, data is often distributed across different organizations or devices (clients) that cannot share raw data due to strict privacy laws or proprietary constraints. For example, different wildlife research stations globally might want to collaborate to discover new bird species without violating local data-sharing regulations.

Figure 1: Illustration of the proposed Fed-GCD with the case of global bird species discovery. In Fed-GCD, the data are distributively collected from the different local stations (clients) over the world, which are partially annotated. Each client includes client-specific categories and may share some common categories with the other clients. Moreover, the raw data stored in local clients are not allowed to share with the central server or other clients, due to data privacy. The goal of Fed-GCD is to collaboratively train a generic GCD model under the federated privacy constraint, and then utilize it to discover novel categories in the unlabeled data on clients or the server during testing.

This paper introduces Federated GCD (Fed-GCD) to address this gap. Fed-GCD introduces two major bottlenecks:

  1. Representation Degradation: Individual clients train on much smaller, localized datasets, harming the model's ability to learn robust features.
  2. Label Space Heterogeneity: The categories present on each client can be highly inconsistent, meaning some clients might hold data for completely non-overlapping sets of classes.

Methods

To tackle these challenges, the authors present the Associated Gaussian Contrastive Learning (AGCL) framework, which moves away from standard instance-level or single-prototype contrastive learning. Instead, it models categories using learnable Gaussian Mixture Models (GMMs).

Figure 2: Diagram of the proposed federated Gaussian contrastive learning (FGCL) framework. We first apply FedAvg McMahan et al. (2017) to aggregate the uploaded local models, resulting in a global model that will be distributed to all clients. Then, after leveraging the distributed models to extract image features, local clients are required to cluster these features and initialize local GMMs. Next, the local GMMs are uploaded to the central server, and aggregated by the proposed CSA, to generate a global GMM before local training. Later, the server distributes the global GMM to each client. Based on the global-local GMMs, client models are collaboratively optimized by the proposed GCL. Finally, a generic model is trained for global category discovery.

The framework operates via three core components:

  • Semi-FINCH for Local Initialization: Clients use an enhanced, semi-supervised version of the parameter-free FINCH clustering algorithm to assign pseudo-labels to local unlabeled data. This allows them to estimate the initial mean and covariance for their local GMMs.
  • Client Semantics Association (CSA): Local GMM parameters are uploaded to a central server. The server samples feature-level instances from these distributions and clusters them to form a comprehensive global GMM. This process implicitly aligns shared classes across clients and maps out broader inter-class relationships.
  • Global-Local GMM Contrastive Learning (GCL): The global GMM is sent back to the clients. Clients then optimize their local models using a combined loss function that contrasts instances against both local and global GMM distributions. A Mahalanobis distance metric is used to dynamically adjust contrastive temperatures based on cluster variances, making the model resilient to noisy pseudo-labels.

Datasets

The authors established a new Fed-GCD benchmark by reorganizing six widely used computer vision datasets:

  • Generic Datasets: CIFAR-10, CIFAR-100, and ImageNet-100.
  • Fine-Grained Datasets: CUB-200 (birds), Stanford Cars, and Oxford-IIIT Pet.

To simulate real-world conditions, they used a $\beta$-Dirichlet distribution to partition the data into two distinct evaluation protocols: Normally Heterogeneous (NH, $\beta=0.2$), where a few common classes are shared, and Extremely Heterogeneous (EH, $\beta=0.05$), where clients share virtually no labeled categories.

Key Insights and Ideas

  • GMMs as a Privacy-Preserving Generative Bridge: Instead of transferring raw images or complex generative models, transferring simple GMM parameters (means and covariances) allows the server to synthesize informative feature-level samples. This achieves knowledge aggregation without privacy leaks.
  • Beyond Single-Vector Prototypes: Traditional prototypical contrastive learning represents a class as a single average vector. By using GMMs, AGCL captures class-specific variance. This allows the model to utilize the Mahalanobis distance, effectively down-weighting unreliable, scattered clusters and preventing out-of-distribution outliers from skewing the representation space.
  • Complementary Global-Local Supervision: Local GCL ensures the model remains highly accurate on the client's specific data distribution, while global GCL introduces category relationships that a single client might never observe locally.

Key Results

The proposed AGCL framework consistently outperformed the standard FedAvg baseline across all datasets and heterogeneity settings:

  • Generic Datasets: In the NH setting on CIFAR-100, AGCL achieved a 56.1% accuracy for all classes, outperforming the FedAvg+GCD baseline by 6.5%. Even in the brutal EH setting (no shared labeled classes), AGCL improved performance by 6.9% on CIFAR-100 and 5.7% on ImageNet-100.
  • Fine-Grained Datasets: Fine-grained tasks showed the most significant gains. On the CUB-200 dataset, AGCL outperformed the baseline by 8.9% in the NH setting and 9.8% in the EH setting, proving highly effective at distinguishing subtle differences between sub-categories in a decentralized manner.

Limitations and Future Work

The authors note that the current setup still relies on storing a small set of labeled data on the central server to estimate the total number of potential categories ($k$) during testing. A key direction for future work is to completely relieve this requirement, allowing for a fully decentralized category estimation process that fits even more realistic deployment scenarios.

Summary updated

Abstract

Generalized category discovery (GCD) aims at grouping unlabeled samples from known and unknown classes, given labeled data of known classes. To meet the recent decentralization trend in the community, we introduce a practical yet challenging task, namely Federated GCD (Fed-GCD), where the training data are distributively stored in local clients and cannot be shared among clients. The goal of Fed-GCD is to train a generic GCD model by client collaboration under the privacy-protected constraint. The Fed-GCD leads to two challenges: 1) representation degradation caused by training each client model with fewer data than centralized GCD learning, and 2) highly heterogeneous label spaces across different clients. To this end, we propose a novel Associated Gaussian Contrastive Learning (AGCL) framework based on learnable GMMs, which consists of a Client Semantics Association (CSA) and a global-local GMM Contrastive Learning (GCL). On the server, CSA aggregates the heterogeneous categories of local-client GMMs to generate a global GMM containing more comprehensive category knowledge. On each client, GCL builds class-level contrastive learning with both local and global GMMs. The local GCL learns robust representation with limited local data. The global GCL encourages the model to produce more discriminative representation with the comprehensive category relationships that may not exist in local data. We build a benchmark based on six visual datasets to facilitate the study of Fed-GCD. Extensive experiments show that our AGCL outperforms the FedAvg-based baseline on all datasets.
BibTeX
@inproceedings{pu2023federated,
  author = {Nan Pu and Zhun Zhong and Xinyuan Ji and Nicu Sebe},
  title = {{Federated Generalized Category Discovery}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure