S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment

Sheng Zhang, Muzammal Naseer, Guangyi Chen, Zhiqiang Shen, Salman H. Khan, Kun Zhang, Fahad Shahbaz KhanAAAI 2024 · OpenReview

TL;DR

This paper introduces a new "Realistic Zero-Shot Classification" (RZSC) setting, which challenges existing methods by assuming no labeled data and a vast, uncurated vocabulary. To tackle this, they propose the S3A framework, which uses a novel Cluster-Vote-Prompt-Realign (CVPR) algorithm to iteratively discover and leverage structural semantic information from unlabeled data, achieving significant performance gains over standard approaches.

What Problem Does This Paper Address?

Traditional zero-shot classification (ZSC) methods, while powerful, often rely on assumptions that don't hold in real-world scenarios. These include having access to some labeled data (even if just for "base" classes) or having a "clean" target vocabulary that perfectly or nearly perfectly matches the actual classes in the unseen data. The authors argue these assumptions are impractical. They introduce "Realistic Zero-Shot Classification" (RZSC) as a more challenging and realistic problem: classifying images from an unlabeled dataset using a massive, broad vocabulary (over 20,000 categories) without any prior knowledge of the target classes. This broad vocabulary introduces significant challenges, particularly "alignment confusion," where similar-sounding or related category names make it hard for models like CLIP to distinguish between them, leading to performance degradation.

Methods

The core of the proposed solution is the Self Structural Semantic Alignment (S3A) framework, which operates through a teacher-student learning strategy. The key innovation is the Cluster-Vote-Prompt-Realign (CVPR) algorithm, designed to extract structural semantic information from unlabeled data. CVPR has four main stages:

  1. Clustering: Image embeddings are clustered using K-Means to group visually similar images.
  2. Voting: Each cluster "votes" on potential category names from the large vocabulary based on the proximity of their embeddings to cluster prototypes. This step identifies initial candidate categories.
  3. Prompting: To resolve ambiguities and differentiate fine-grained categories, a Large Language Model (LLM) is used to generate descriptive attributes for candidate category names. These attributes are then incorporated into discriminative prompts (e.g., "A photo of a [category] with [attribute]").
  4. Re-alignment: The LLM-generated prompts are used to refine the alignment between image clusters and vocabulary, creating more robust structural semantic alignment labels.

This CVPR algorithm is run periodically (e.g., per epoch) on an EMA-averaged "teacher" model. The "student" model (the one being trained) then learns from both:

  • Instance-wise pseudo labels: Direct predictions from the teacher model for individual images.
  • Structural semantic alignment labels: The refined alignments generated by the CVPR algorithm.

This teacher-student setup with iterative refinement allows the model to learn from its own generated pseudo-supervision, progressively improving its understanding of image-category relationships within the large vocabulary.

Datasets

The authors evaluate their method on a variety of datasets:

  • Generic Benchmarks: ImageNet-100 (IN100), ImageNet-1K (IN1K).
  • Fine-grained Benchmarks: StanfordDogs (SDogs), Living17 (LV17), Nonliving26 (NL26), Entity13 (ET13), Entity30 (ET30).
  • Out-of-Vocabulary (OOV) Benchmarks: Oxford-IIIT Pet (Pet), CIFAR100, Caltech101. These are specifically used to test how well the method generalizes to categories not present in its training vocabulary.

The large vocabulary used for training and evaluation is derived from ImageNet-1K and ImageNet-21K, encompassing over 20,000 categories.

Key Insights and Ideas

  • The "Realistic" Zero-Shot Challenge: The paper effectively frames a more practical zero-shot problem by removing assumptions about labeled data and vocabulary quality, highlighting a critical gap in current ZSC research.
  • Structural Semantics are Key: The core idea is that beyond just matching individual image-text pairs, understanding the structural relationships between categories (how they group and relate to each other) is crucial for robust classification with large vocabularies.
  • LLMs as Discriminative Augmenters: Using LLMs to generate descriptive attributes for category names is a clever way to inject nuanced semantic information, helping to disambiguate similar categories that confuse standard VLMs.
  • Iterative Self-Improvement: The teacher-student learning with CVPR running periodically creates a powerful self-training loop. The teacher provides increasingly reliable pseudo-labels, which the student learns from, and in turn, the student's improved representations help the teacher generate even better pseudo-labels in the next CVPR iteration.
  • Robustness to Vocabulary Size: The significant performance drop of CLIP with a large vocabulary is clearly demonstrated, underscoring the need for methods like S3A that can handle this challenge.

Key Results

  • State-of-the-Art Performance: S3A consistently achieves state-of-the-art (SOTA) results on the proposed RZSC setting across multiple generic and fine-grained benchmarks.
  • Significant Accuracy Gains: The method improves accuracy by over 15% compared to CLIP on average. It also outperforms adapted SOTA baselines by nearly 5% in top-1 accuracy and 6% in clustering accuracy.
  • Out-of-Vocabulary Generalization: S3A demonstrates strong performance even on OOV benchmarks, indicating its ability to generalize beyond its explicit training vocabulary.
  • Effectiveness of CVPR Components: Ablation studies show that all components of CVPR (clustering, voting, prompting, re-alignment) and the self-training strategy contribute to the overall performance improvement. The LLM-based prompt augmentation, in particular, is shown to be crucial for semantic discrimination.
  • Robustness to K Estimation: The method shows resilience to inaccuracies in estimating the number of classes (K), with only a marginal performance drop when K is slightly off.

Limitations and Future Work

  • Computational Cost: While effective, the iterative nature of CVPR and the use of LLMs for prompt augmentation likely incur significant computational costs, especially during training.
  • LLM Dependency: The performance of the prompt augmentation step is dependent on the capabilities of the LLM used.
  • Future Work: The authors suggest exploring more efficient ways to estimate K and potentially integrating more advanced LLM-based prompt generation techniques. Further research into handling even larger and more dynamic vocabularies could also be a direction. The paper also mentions exploring the impact of different backbone architectures and further optimizing the self-training process.

Summary updated

Abstract

Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. Despite the success, most traditional VLMs-based methods are restricted by the assumption of partial source supervision or ideal target vocabularies, which rarely satisfy the open-world scenario. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. To address the new problem, we propose the Self Structural Semantic Alignment (S3A) framework, which extracts the structural semantic information from unlabeled data while simultaneously self-learning. Our S3A framework adopts a unique Cluster-Vote-Prompt-Realign (CVPR) algorithm, which iteratively groups unlabeled data to derive structural semantics for pseudo-supervision. Our CVPR algorithm includes iterative clustering on images, voting within each cluster to identify initial class candidates from the vocabulary, generating discriminative prompts with large language models to discern confusing candidates, and realigning images and the vocabulary as structural semantic alignment. Finally, we propose to self-train the CLIP image encoder with both individual and structural semantic alignment through a teacher-student learning strategy. Our comprehensive experiments across various generic and fine-grained benchmarks demonstrate that the S3A method substantially improves over existing VLMs-based approaches, achieving a more than 15% accuracy improvement over CLIP on average. Our codes, models, and prompts are publicly released at https://github.com/sheng-eatamath/S3A.
BibTeX
@inproceedings{zhang2024s,
  author = {Sheng Zhang and Muzammal Naseer and Guangyi Chen and Zhiqiang Shen and Salman H. Khan and Kun Zhang and Fahad Shahbaz Khan},
  title = {{S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=ro5B9T6qa6},
}
Figure