VLM-NCD:Novel Class Discovery with Vision-Based Large Language Models

Yuetong Su, Bao-guo Wei, Xinyu Wang, Li Xu, Lixin LiarXiv 2025 · DOI · arXiv

Abstract

Novel Class Discovery aims to utilise prior knowledge of known classes to classify and discover unknown classes from unlabelled data. Existing NCD methods for images primarily rely on visual features, which suffer from limitations such as insufficient feature discriminability and the long-tail distribution of data. We propose LLM-NCD, a multimodal framework that breaks this bottleneck by fusing visual-textual semantics and prototype guided clustering. Our key innovation lies in modelling cluster centres and semantic prototypes of known classes by jointly optimising known class image and text features, and a dualphase discovery mechanism that dynamically separates known or novel samples via semantic affinity thresholds and adaptive clustering. Experiments on the CIFAR-100 dataset show that compared to the current methods, this method achieves up to 25.3% improvement in accuracy for unknown classes. Notably, our method shows unique resilience to long tail distributions, a first in NCD literature.

Introduction

Novel Class Discovery (NCD) is a pivotal task in open-world recognition, where AI systems must adapt to new categories not encountered during initial training. The task involves leveraging a set of labeled "known" classes to guide the identification and clustering of "unknown" classes within an unlabeled dataset.

Traditional NCD methods rely almost exclusively on deep visual features. However, this vision-only approach faces three critical bottlenecks: insufficient feature discrimination for new categories, bias in long-tail distributions where rare classes are often misclustered, and a lack of multimodal context that could provide essential semantic clues. To address these, we present LLM-NCD, a framework that leverages Large Language Models (LLMs) and multimodal embeddings to enhance discovery performance.

Methodology

The LLM-NCD framework moves beyond pixels by incorporating external textual knowledge into the discovery process.

1. Retrieval-Based Data Augmentation

Since novel class labels are unknown, the framework uses input images as queries to retrieve the top-$k$ most relevant text descriptions from massive external knowledge databases like CC-12M or LAION. This provides necessary background knowledge without requiring predefined labels.

2. Multimodal Feature Fusion

Utilizing a frozen CLIP model, the system encodes images and retrieved text descriptions separately. These features are then fused—specifically, text features are mean-pooled and concatenated with image features—to create a comprehensive "Feature Code." This joint representation benefits from both visual granularity and textual semantic constraints.

Detailed architecture of the LLM-NCD framework, illustrating the fusion of visual and textual features for clustering.

3. Semi-Supervised Clustering

The model employs a dual-phase discovery mechanism. It first establishes stable cluster centers using labeled data. In the discovery stage, a semi-supervised k-means algorithm is used to cluster unlabeled samples, forcing labeled samples into correct clusters while allowing unlabeled samples to find semantically similar neighbors.

Experimental Results

LLM-NCD was evaluated on CIFAR-10, CIFAR-100, and ImageNet-100 benchmarks. The results demonstrate a significant advancement over previous state-of-the-art (SOTA) methods like GCD and XCon. On CIFAR-100, the framework achieved an 85.6% accuracy on novel classes, representing a 25.3% improvement over the previous SOTA.

Impact of Retrieval Depth

A key insight from the experiments is the existence of a "sweet spot" for text retrieval. As shown in the analysis of Top-k values, retrieving approximately four descriptions per image yields the highest accuracy. While too few descriptions provide insufficient context, an excessive number (e.g., $k=8$) introduces noise that can degrade clustering performance.

Accuracy performance across different Top-k retrieval values, highlighting the optimal balance at k=4.

Conclusion

LLM-NCD significantly advances the field of Novel Class Discovery by integrating visual features with retrieved textual semantics. By acting as a semantic anchor, text descriptions provide unprecedented resilience to long-tail data distributions. Future research will focus on improving retrieval quality for niche domains and automating the estimation of the number of novel classes.

Summary updated

Abstract

Novel Class Discovery aims to utilise prior knowledge of known classes to classify and discover unknown classes from unlabelled data. Existing NCD methods for images primarily rely on visual features, which suffer from limitations such as insufficient feature discriminability and the long-tail distribution of data. We propose LLM-NCD, a multimodal framework that breaks this bottleneck by fusing visual-textual semantics and prototype guided clustering. Our key innovation lies in modelling cluster centres and semantic prototypes of known classes by jointly optimising known class image and text features, and a dualphase discovery mechanism that dynamically separates known or novel samples via semantic affinity thresholds and adaptive clustering. Experiments on the CIFAR-100 dataset show that compared to the current methods, this method achieves up to 25.3% improvement in accuracy for unknown classes. Notably, our method shows unique resilience to long tail distributions, a first in NCD literature.
BibTeX
@misc{su2025vlm,
  author = {Yuetong Su and Bao-guo WEI and Xinyu Wang and Li Xu and Lixin Li},
  title = {{VLM-NCD:Novel Class Discovery with Vision-Based Large Language Models}},
  year = {2025},
  eprint = {2512.10262},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.10262},
  howpublished = {arXiv},
}
Figure