TL;DR
OmniGCD introduces a modality-agnostic, zero-shot framework for Generalized Category Discovery (GCD), mimicking the human brain's ability to categorize abstract concepts independently of sensory inputs. By training a specialized Transformer model (GCDformer) just once on purely synthetic data, OmniGCD successfully clusters both known and novel categories across vision, text, audio, and remote sensing datasets without any dataset-specific fine-tuning.
What Problem Does This Paper Address?
Generalized Category Discovery (GCD) is the task of recognizing known categories while simultaneously clustering novel, unseen categories using partially labeled data. While highly reflective of how humans learn, existing machine learning approaches for GCD suffer from two major flaws:
- Modality Siloing: Methods are strictly locked into a single domain (e.g., only computer vision or only natural language processing).
- Heavy Fine-Tuning: They require independent, computationally expensive fine-tuning on every single dataset they are evaluated on.
In contrast, the human brain handles categorization abstractly in the prefrontal cortex, integrating highly preprocessed signals regardless of whether they come from our eyes, ears, or other senses. OmniGCD addresses this gap by decoupling representation learning (the sensory encoders) from category discovery (the abstract reasoning), creating a single model capable of zero-shot category discovery across entirely different data modalities.
Methods
OmniGCD splits the problem into two distinct stages: representation extraction and geometric latent space optimization.
- Tokenization & Dimension Reduction: Features are extracted using frozen, off-the-shelf foundation models specific to each modality. To make the geometric space tractable for a universal model, these high-dimensional features are projected into a low-dimensional (2D) space using t-SNE.
- The GCD Latent Space: These 2D coordinates become "Data Tokens." For labeled data, they are concatenated with a sinusoidal "Label Embedding Token." For unlabeled data, they are paired with a learnable "Masked Label Token."
- GCDformer: The core architecture is a non-causal Transformer (based on GPT-2) that processes these tokens as a set. It uses self-attention to look at the global distribution of points and outputs an optimized latent space where clusters are tighter and more distinct.
- Synthetic Training: Crucially, GCDformer is trained only once on synthetically generated cluster data (using Gaussian, Laplace, uniform, and von Mises distributions). This ensures the model learns pure geometric clustering principles rather than overfitting to any specific real-world data modality.
- Clustering: At test time, standard $k$-means clustering is applied to the optimized latent space to produce final classifications.
Datasets
The authors evaluate OmniGCD in a strict zero-shot setting across 16 distinct datasets spanning four modalities:
- Vision: CIFAR-10, CIFAR-100, ImageNet-100, CUB-200, Stanford Cars, FGVC-Aircraft, Herbarium-19.
- Text: BANKING, StackOverflow, CLINIC.
- Audio: VocalSet, UrbanSound.
- Remote Sensing (Hyperspectral/Satellite): EuroSAT, So2SAT, RESISC45, UC Merced.
Key Insights and Ideas
- Decoupling is Key: By separating encoder training from category discovery, OmniGCD can seamlessly inherit the benefits of future foundation models. If someone trains a better vision or audio encoder tomorrow, OmniGCD can use it immediately without retraining the discovery module.
- The Power of Synthetic Geometry: A model doesn't need to understand semantic real-world concepts to cluster them. Training a Transformer on abstract mathematical distributions (like Gaussians with random masking) is sufficient to teach it how to fix messy, real-world feature spaces for clustering.
- t-SNE Outperforms Linear Methods: For low-dimensional mapping, non-linear dimension reduction techniques like t-SNE preserve local neighborhood structures much better than linear methods like PCA, leading to significantly better synthetic-to-real alignment.
Key Results
OmniGCD consistently outperforms baseline methods (like raw $k$-means or the original GCD framework without fine-tuning) across all four modalities:
- Vision: Achieved an average +6.2 percentage point (pp) accuracy improvement over baselines using DINOv1 and DINOv2 encoders.
- Text: Showed the most dramatic gains, with an average +17.9 pp boost in overall accuracy.
- Audio: Marked the first-ever application of GCD to the audio modality, yielding a +1.5 pp average improvement, driven heavily by massive gains in discovering novel classes (+23.2 pp on VocalSet and +26.2 pp on UrbanSound).
- Remote Sensing: Improved baseline performance by an average of +12.7 pp.
Limitations and Future Work
- Dependency on Encoder Quality: Because OmniGCD freezes the underlying encoders, it is bound by the "garbage in, garbage out" rule. If an encoder produces poor representations (as seen with DINOv1 on fine-grained datasets like Stanford Cars), OmniGCD's performance degrades.
- Fine-Tuning Gap: On highly specialized, fine-grained datasets, traditional methods that fine-tune the encoder on the target dataset still hold an advantage.
- Future Directions: The authors suggest exploring end-to-end training frameworks for OmniGCD rather than optimizing the GCDformer in isolation, which could bridge the performance gap on fine-grained tasks.
Summary updated