OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism

Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, Clinton FookesCVPR 2026 Findings 2026 · OpenReview

TL;DR

OmniGCD introduces a modality-agnostic, zero-shot framework for Generalized Category Discovery (GCD), mimicking the human brain's ability to categorize abstract concepts independently of sensory inputs. By training a specialized Transformer model (GCDformer) just once on purely synthetic data, OmniGCD successfully clusters both known and novel categories across vision, text, audio, and remote sensing datasets without any dataset-specific fine-tuning.

What Problem Does This Paper Address?

Generalized Category Discovery (GCD) is the task of recognizing known categories while simultaneously clustering novel, unseen categories using partially labeled data. While highly reflective of how humans learn, existing machine learning approaches for GCD suffer from two major flaws:

  1. Modality Siloing: Methods are strictly locked into a single domain (e.g., only computer vision or only natural language processing).
  2. Heavy Fine-Tuning: They require independent, computationally expensive fine-tuning on every single dataset they are evaluated on.

In contrast, the human brain handles categorization abstractly in the prefrontal cortex, integrating highly preprocessed signals regardless of whether they come from our eyes, ears, or other senses. OmniGCD addresses this gap by decoupling representation learning (the sensory encoders) from category discovery (the abstract reasoning), creating a single model capable of zero-shot category discovery across entirely different data modalities.

Summary of OmniGCD for modality-agnostic zero-shot GCD, inspired by human category learning.

Methods

OmniGCD splits the problem into two distinct stages: representation extraction and geometric latent space optimization.

  1. Tokenization & Dimension Reduction: Features are extracted using frozen, off-the-shelf foundation models specific to each modality. To make the geometric space tractable for a universal model, these high-dimensional features are projected into a low-dimensional (2D) space using t-SNE.
  2. The GCD Latent Space: These 2D coordinates become "Data Tokens." For labeled data, they are concatenated with a sinusoidal "Label Embedding Token." For unlabeled data, they are paired with a learnable "Masked Label Token."
  3. GCDformer: The core architecture is a non-causal Transformer (based on GPT-2) that processes these tokens as a set. It uses self-attention to look at the global distribution of points and outputs an optimized latent space where clusters are tighter and more distinct.
  4. Synthetic Training: Crucially, GCDformer is trained only once on synthetically generated cluster data (using Gaussian, Laplace, uniform, and von Mises distributions). This ensures the model learns pure geometric clustering principles rather than overfitting to any specific real-world data modality.
  5. Clustering: At test time, standard $k$-means clustering is applied to the optimized latent space to produce final classifications.
Figure 2. Overview diagram of OmniGCD. Data tokens are pro- duced from images of the labeled and unlabeled datasets. The data tokens are then mapped to a GCD latent space and concatenated (shown as ++) with either label embedding or masked label tokens. The GCD latent space is then transformed at test-time into a rep- resentation better suited for clustering by GCDformer, before final classifications are produced by k-means clustering.

Datasets

The authors evaluate OmniGCD in a strict zero-shot setting across 16 distinct datasets spanning four modalities:

  • Vision: CIFAR-10, CIFAR-100, ImageNet-100, CUB-200, Stanford Cars, FGVC-Aircraft, Herbarium-19.
  • Text: BANKING, StackOverflow, CLINIC.
  • Audio: VocalSet, UrbanSound.
  • Remote Sensing (Hyperspectral/Satellite): EuroSAT, So2SAT, RESISC45, UC Merced.

Key Insights and Ideas

  • Decoupling is Key: By separating encoder training from category discovery, OmniGCD can seamlessly inherit the benefits of future foundation models. If someone trains a better vision or audio encoder tomorrow, OmniGCD can use it immediately without retraining the discovery module.
  • The Power of Synthetic Geometry: A model doesn't need to understand semantic real-world concepts to cluster them. Training a Transformer on abstract mathematical distributions (like Gaussians with random masking) is sufficient to teach it how to fix messy, real-world feature spaces for clustering.
  • t-SNE Outperforms Linear Methods: For low-dimensional mapping, non-linear dimension reduction techniques like t-SNE preserve local neighborhood structures much better than linear methods like PCA, leading to significantly better synthetic-to-real alignment.

Key Results

OmniGCD consistently outperforms baseline methods (like raw $k$-means or the original GCD framework without fine-tuning) across all four modalities:

  • Vision: Achieved an average +6.2 percentage point (pp) accuracy improvement over baselines using DINOv1 and DINOv2 encoders.
  • Text: Showed the most dramatic gains, with an average +17.9 pp boost in overall accuracy.
  • Audio: Marked the first-ever application of GCD to the audio modality, yielding a +1.5 pp average improvement, driven heavily by massive gains in discovering novel classes (+23.2 pp on VocalSet and +26.2 pp on UrbanSound).
  • Remote Sensing: Improved baseline performance by an average of +12.7 pp.
Zero-shot GCD results across all modalities.

Limitations and Future Work

  • Dependency on Encoder Quality: Because OmniGCD freezes the underlying encoders, it is bound by the "garbage in, garbage out" rule. If an encoder produces poor representations (as seen with DINOv1 on fine-grained datasets like Stanford Cars), OmniGCD's performance degrades.
  • Fine-Tuning Gap: On highly specialized, fine-grained datasets, traditional methods that fine-tune the encoder on the target dataset still hold an advantage.
  • Future Directions: The authors suggest exploring end-to-end training frameworks for OmniGCD rather than optimizing the GCDformer in isolation, which could bridge the performance gap on fine-grained tasks.

Summary updated

Abstract

Generalized Category Discovery (GCD) challenges methods to identify known and novel classes using partially labeled data, mirroring human category learning. Unlike prior GCD methods, which operate within a single modality and require dataset-specific fine-tuning, we propose a modality-agnostic GCD approach inspired by the human brain's abstract category formation. Our OmniGCD leverages modality-specific encoders (e.g., vision, audio, text, remote sensing) to process inputs, followed by dimension reduction to construct a GCD latent space, which is transformed at test-time into a representation better suited for clustering using a novel synthetically trained Transformer-based model. To evaluate OmniGCD, we introduce a zero-shot GCD setting where no dataset-specific fine-tuning is allowed, enabling modality-agnostic category discovery. Trained once on synthetic data, OmniGCD performs zero-shot GCD across 16 datasets spanning four modalities, improving classification accuracy for known and novel classes over baselines (average percentage point improvement of +6.2, +17.9, +1.5 and +12.7 for vision, text, audio and remote sensing). This highlights the importance of strong encoders while decoupling representation learning from category discovery. Improving modality-agnostic methods will propagate across modalities, enabling encoder development independent of GCD. Our work serves as a benchmark for future modality-agnostic GCD works, paving the way for scalable, human-inspired category discovery.
BibTeX
@inproceedings{shipard2026omnigcd,
  author = {Jordan Shipard and Arnold Wiliem and Kien Nguyen Thanh and Wei Xiang and Clinton Fookes},
  title = {{OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=vHyPzh8BP9},
}
Figure