Abstract
Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. While such capability is largely attributed to the rich world knowledge contained within the Large Language Models (LLMs), our work reveals their shortcomings in fine-grained visual categorization (FGVC) across six different benchmark settings. Most recent state-of-the-art LVLMs such as LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs for LLaVA-1.5, but also struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. In-depth analyses show that instruction-tuned LVLMs suffer from modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept. In an effort to further the community’s endeavor in this direction, we propose a multiple granularity attribute-centric benchmark and training mixture, Finer, which aims to establish a ground to evaluate LVLMs’ fine-grained visual comprehension ability and provide significantly improved explainability.
Introduction: The Paradox of Visual Knowledge
Large Vision-Language Models (LVLMs) like GPT-4V and LLaVA have demonstrated an uncanny ability to generate poetic, high-level descriptions of complex scenes. However, a significant paradox exists: while these models can describe a scene with nuance, they often fail at the fundamental task of identifying specific species or technical models. For instance, a model might correctly describe a bird as having a "prominent white beak and a patterned head," yet fail to identify it as a "Kakapo," defaulting instead to the generic label of "parrot."
This research identifies a critical "modality gap" in current state-of-the-art models. While the underlying Large Language Model (LLM) backbones possess the world knowledge to describe a "Kakapo" in text, the visual features are often diluted or lost during the projection from raw pixels to tokens. This makes LVLMs brittle for specialized tasks like Fine-Grained Visual Categorization (FGVC), which is essential for fields ranging from biological research to technical inspection.
The Modality Gap and the Projector Bottleneck
The authors hypothesize that the discrepancy between textual and visual processing stems from the "projector" layer—the bridge between the vision encoder and the LLM. Linear probing analysis revealed that while the vision encoder itself captures fine-grained details, the projection layer acts as a bottleneck where this specific information is discarded in favor of more general semantic tokens. Consequently, even if a model "knows" the concept in its text-based memory, it cannot recognize it visually.
The FINER Framework: ATTRSEEK and Instruction Tuning
To address these shortcomings, the researchers proposed the FINER framework, which consists of two primary interventions:
- ATTRSEEK (Prompting Technique): This two-stage prompting strategy forces the model to attend to visual details. First, the model is prompted to generate a list of distinctive physical attributes visible in the image (e.g., "T-tail design," "rear-mounted engines"). Second, it uses these self-generated attributes as context to reach a final fine-grained classification.
- FINER Training Mixture: The authors developed a massive instruction-tuning dataset to fine-tune models like LLaVA-1.5. This training forces the model to follow the ATTRSEEK pipeline, explicitly teaching it to associate fine-grained visual details with specific labels.
Experimental Evaluation and Results
The study utilized six classic FGVC benchmarks, including iNaturalist-2021, NABirds, and Stanford Cars. The researchers introduced multi-granular labels—Superordinate (e.g., Bird), Coarse (e.g., Parrot), and Fine (e.g., Kakapo)—to measure performance decay as specificity increases.
The results highlight a severe performance collapse at the fine-grained level. On the iNaturalist-2021 dataset, LLaVA-1.5 (13B) achieves nearly 97% accuracy at the Superordinate level but drops to approximately 1.5% at the Fine level. However, the proposed interventions yielded significant gains:
- GPT-4V: Using the ATTRSEEK strategy, fine-grained classification accuracy on iNaturalist jumped from 18.75% to 53.12% (Exact Match).
- LLaVA-1.5: After fine-tuning on the FINER mixture, performance on the Stanford Dogs dataset more than doubled, rising from 16.7% to 36.3%.
Conclusion and Future Work
The FINER benchmark demonstrates that the current bottleneck in visual recognition is not a lack of knowledge in the LLM, but a failure in the vision-to-language bridge. By emphasizing attribute-centric reasoning, the researchers have provided a path toward more explainable and accurate LVLMs. Future work will focus on addressing intra-concept variance—such as how a species looks different across various life stages—and improving the architectural design of the projection layer to preserve high-fidelity visual information.
Summary updated