Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models

Canshi WeiEMNLP 2024 · DOI

Abstract

Fine-grained image classification requires distinguishing between highly similar sub-categories, a task where standard Vision Language Models (VLMs) often struggle due to a lack of granular depth. While Large Vision Language Models (LVLMs) possess extensive world knowledge, they are hindered by positional bias and high inference costs. This paper introduces CascadeVLM, a framework that leverages CLIP-like models to filter and rank candidates before passing them to an LVLM. By using adaptive entropy thresholding to gate samples, CascadeVLM achieves state-of-the-art performance on benchmarks like Stanford Cars and iNaturalist 2018 while maintaining computational efficiency.

Introduction: The Challenge of Fine-Grained Classification

Fine-grained classification involves identifying subtle differences, such as distinguishing between specific car model years or rare biological species. Standard VLMs like CLIP provide strong zero-shot baselines but often lack the granular supervision needed for these subtle distinctions. Conversely, LVLMs like GPT-4V face two primary hurdles:

  1. Positional Bias: When presented with long lists of categories, LVLMs tend to favor options appearing early in the text.
  2. Computational Cost: The scale of models like GPT-4V makes running them on every image in a large dataset prohibitively expensive.

Methodology: The CascadeVLM Framework

CascadeVLM addresses these issues through a multi-stage pipeline designed to optimize both accuracy and efficiency by playing to the strengths of both model types.

The CascadeVLM pipeline: CLIP ranks candidates, entropy gates the process, and LVLMs provide the final expert reasoning.

The process consists of four main steps:

  1. Candidate Selection: A CLIP backbone scores all categories and extracts the top-$k$ candidates, which are then sorted in descending order of probability.
  2. Adaptive Entropy Thresholding: The system calculates the entropy (uncertainty) of CLIP's prediction. If CLIP is confident (low entropy), it makes the final call. If uncertain (high entropy), the image and narrowed list are passed to the LVLM.
  3. LVLM Reasoning: The LVLM receives the image and the pre-sorted list of candidates. This setup turns the LVLM's positional bias into an advantage by placing the most likely candidates where the model focuses most.
  4. Iterative Refinement: In specific configurations, the LVLM can request additional candidates if the initial top-$k$ set does not contain a suitable match.

Experimental Evaluation

The framework was evaluated on five prominent fine-grained benchmarks: Flowers102, Stanford Cars (196 classes), FGVC Aircraft, BirdSnap (500 species), and the massive iNaturalist 2018 (8,142 classes).

Key Results

CascadeVLM consistently outperformed standalone models and existing prompt-engineering methods like CoOp and CoCoOp.

  • Stanford Cars: Achieved a peak accuracy of 92.0% in a few-shot setting using Gemini-1.5-Pro.
  • iNaturalist 2018: This complex dataset saw the most dramatic improvements. While a standalone MAWS-CLIP achieved ~21% accuracy, cascading it with GPT-4o boosted performance to 29%.
Performance comparison on the iNaturalist 2018 dataset, showing significant accuracy gains using the CascadeVLM framework with Gemini and GPT-4o.

Discussion and Insights

The core innovation of CascadeVLM lies in "turning a bug into a feature." By sorting candidates by probability, the framework aligns the most likely answers with the LVLM's inherent positional bias. Furthermore, the entropy gate allows for significant cost savings; the authors demonstrated that a portion of samples can be handled by CLIP alone without losing significant accuracy, drastically reducing expensive API calls.

Limitations and Future Work

Despite its success, the framework is bounded by the "Top-K" ceiling—if the correct class is not in CLIP's initial selection, the LVLM may fail unless iterative refinement is used. Additionally, while the entropy gate reduces costs, the system remains more resource-intensive than single-pass VLM models. Future work may involve using smaller, distilled LVLMs to maintain the cascade benefit at a lower cost.

Conclusion

CascadeVLM represents a significant step forward in fine-grained classification by effectively combining the efficiency of VLMs with the reasoning power of LVLMs. It provides a scalable solution for complex visual tasks where granular world knowledge is paramount.

Summary updated

BibTeX
@inproceedings{wei2024enhancing,
  author = {Canshi Wei},
  title = {{Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.102},
}
Figure