Federated Generalized Category Discovery

Nan Pu, Wenjing Li, Xingyuan Ji, Yalan Qin, Nicu Sebe, Zhun Zhong

Abstract: Federated Generalized Category Discovery (Fed-GCD) is a novel task that requires decentralized clients to categorize both known and unknown classes without sharing raw data. This paper introduces Associated Gaussian Contrastive Learning (AGCL), a framework utilizing learnable Gaussian Mixture Models (GMMs) to bridge client gaps. By aggregating class-level variance and mean through GMMs, AGCL enables robust feature learning and global category alignment. Experiments on generic and fine-grained datasets demonstrate that AGCL significantly outperforms standard federated baselines.

What Problem Does This Paper Address?

In computer vision, Generalized Category Discovery (GCD) is often considered a "holy grail" of classification. The goal is to take a set of unlabeled samples and group them into both known classes (from a small labeled set) and entirely new, unknown categories.

Traditionally, GCD research has assumed a centralized environment where all data is accessible on a single server. However, real-world data—such as medical records or ecological observations—is frequently distributed across different institutions (clients) that cannot share raw data due to privacy regulations. This decentralized reality introduces two major hurdles:

  1. Representation Degradation: Individual clients often possess limited data, making it difficult for models to learn robust, generalizable features.
  2. Label Heterogeneity: Different clients observe different sets of categories based on their location or specialization (e.g., a bird-watching station in Asia sees different species than one in Europe). Building a unified global model under these conditions is exceptionally difficult.
Overview of the Federated Generalized Category Discovery (Fed-GCD) task.

Methods: Associated Gaussian Contrastive Learning (AGCL)

To address these challenges, the authors propose Associated Gaussian Contrastive Learning (AGCL). Unlike standard federated learning (like FedAvg) which simply averages model weights, AGCL focuses on capturing and aligning class-level relationships through a more sophisticated statistical lens.

The Associated Gaussian Contrastive Learning (AGCL) framework architecture.

The AGCL framework consists of several key components designed to work in concert across the client-server boundary:

  • Learnable GMMs: Rather than representing a class as a single point (a prototype), AGCL models each class as a Gaussian Mixture Model (GMM). This approach accounts for the variance and "spread" of a class in the feature space. By using Mahalanobis distance, the model performs more robust, class-aware contrastive learning that is less sensitive to outliers.
  • Client Semantics Association (CSA): The central server aggregates the GMM parameters (means and variances) uploaded by clients. By sampling feature-level instances from these GMMs and re-clustering them, the server constructs a "Global GMM." This aligns shared classes across different clients and builds a comprehensive knowledge base without the server ever seeing a raw image.
  • Global-Local GMM Contrastive Learning (GCL): Once the global GMM is sent back to the clients, the local models perform contrastive learning using both their local GMM and the server’s global view. This forces local models to respect global category relationships, effectively providing "supervision" for classes the client might not have in its own dataset.
  • Semi-FINCH: To initialize these GMMs without a predefined number of classes, the authors developed a semi-supervised version of the FINCH clustering algorithm, allowing for flexible discovery of unknown categories.

Datasets and Experimental Setup

The authors established a new benchmark for Fed-GCD by reorganizing six popular datasets into federated partitions:

  • Generic Datasets: CIFAR-10, CIFAR-100, and ImageNet-100.
  • Fine-Grained Datasets: CUB-200 (birds), Stanford Cars, and Oxford-IIIT Pet.

To simulate realistic data distributions, they employed a Dirichlet distribution (controlled by parameter $\beta$) to split the data. This created scenarios ranging from "normally heterogeneous" (some shared classes) to "extremely heterogeneous" (almost no shared classes between clients), testing the model's ability to generalize in the toughest conditions.

Key Insights and Results

The AGCL framework consistently outperformed the standard FedAvg + GCD baseline across all benchmarks:

  • Variance Matters: Traditional prototypical learning treats every class as a uniform sphere. By using GMMs, AGCL acknowledges that some classes are more diverse than others, making the model significantly more robust to noisy or incorrectly clustered samples.
  • Feature-Level Collaboration: The research demonstrates that raw images are not necessary for effective collaboration. Sharing GMM parameters allows clients to "describe" their classes to the server, which then synthesizes a global perspective that benefits all participants.
  • Performance Gains: On CIFAR-100, AGCL improved accuracy by 6.5% to 6.9%. The gains were even more significant in fine-grained tasks like the CUB-200 bird dataset, where AGCL outperformed the baseline by 8.9% to 9.8%.

Limitations and Future Work

While AGCL represents a major advancement, the authors identified several areas for future research. Currently, the method relies on a ViT backbone pre-trained on ImageNet; future work could explore training such models from scratch in a federated environment. Additionally, the current setup assumes the server has access to some labeled data for "Old" classes to facilitate testing. Moving toward a scenario where the server requires even less labeled data remains a goal. Finally, the communication overhead of frequent GMM updates in massive-scale deployments is an open question for optimization.

Summary updated

Abstract

Generalized category discovery (GCD) aims at grouping unlabeled samples from known and unknown classes, given labeled data of known classes. To meet the recent decen-tralization trend in the community, we introduce a practical yet challenging task, Federated GCD (Fed-GCD), where the training data are distributed among local clients and cannot be shared among clients. Fed-GCD aims to train a generic GCD model by client collaboration under the privacy-protected constraint. The Fed-GCD leads to two challenges: 1) representation degradation caused by training each client model with fewer data than centralized GCD learning, and 2) highly heterogeneous label spaces across different clients. To this end, we propose a novel Asso-ciated Gaussian Contrastive Learning (AGCL) framework based on learnable GMMs, which consists of a Client Se-mantics Association (CSA) and a global-local GMM Contrastive Learning (GCL). On the server, CSA aggregates the heterogeneous categories of local-client GMMs to generate a global GMM containing more comprehensive category knowledge. On each client, GCL builds class-level contrastive learning with both local and global GMMs. The local GCL learns robust representation with limited local data. The global GCL encourages the model to produce more discriminative representation with the comprehensive category relationships that may not exist in local data. We build a benchmark based on six visual datasets to facilitate the study of Fed-GCD. Extensive experiments show that our AGCL outperforms multiple baselines on all datasets. Code is available at https://github.com/TPCD/FedGCD.
BibTeX
@inproceedings{pu2023federated,
  author = {Nan Pu and Wenjing Li and Xingyuan Ji and Yalan Qin and Nicu Sebe and Zhun Zhong},
  title = {{Federated Generalized Category Discovery}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure