TL;DR
This paper introduces VLM4Bio, a new benchmark dataset designed to evaluate the capabilities of large vision-language models (VLMs) in organismal biology. They found that while VLMs show promise, they struggle with complex biological reasoning and localization tasks, highlighting a significant gap between current VLM performance and the needs of scientific discovery in this field.
What Problem Does This Paper Address?
The explosion of biodiversity data, particularly in the form of images, presents a massive opportunity for scientific discovery in organismal biology. Traditionally, extracting biological traits from these images has been a labor-intensive process requiring expert knowledge. The advent of powerful Vision-Language Models (VLMs) offers a potential solution to automate this process. However, existing VLMs are often trained on general datasets and may lack the domain-specific knowledge and reasoning abilities required for biological tasks. This paper addresses the critical need for a specialized benchmark to evaluate how well current VLMs can understand and reason about biological images, specifically for trait discovery, and to identify their limitations in this domain.
Methods
The authors developed the VLM4Bio dataset, comprising approximately 469,000 question-answer pairs derived from 30,000 images of fishes, birds, and butterflies. They designed five biologically relevant tasks: species classification, trait identification, trait grounding (locating a trait), trait referring (identifying a trait from a location), and trait counting. To evaluate VLMs, they employed a zero-shot approach, meaning the models were tested without any specific fine-tuning on the VLM4Bio dataset. They also investigated the impact of different prompting techniques (contextual, dense captioning, and chain-of-thought) and reasoning hallucination tests (False Confidence Test and None of the Above Test) on VLM performance.
Datasets
The primary contribution is the VLM4Bio dataset, which includes:
- Images: Approximately 30,000 images of fishes (Fish-10K), birds (Bird-10K), and butterflies (Butterfly-10K).
- Question-Answer Pairs: Around 469,000 pairs covering five tasks.
- Existing Datasets Used:
- FishAIR [24] and Integrated Digitized Biocollections (iDigBio) [26] for fish images.
- Jiggins Heliconius Collection dataset [35] for butterfly images.
- CUB-200-2011 [61] for bird images.
VLM4Bio differs from existing benchmarks by focusing specifically on organismal biology and offering a diverse set of biologically relevant tasks that require both predictive accuracy and visual reasoning.
Key Insights and Ideas
- VLMs struggle with open-ended biological questions: While VLMs perform better on multiple-choice questions, their accuracy drops significantly when asked to provide answers without options, indicating a gap in their ability to recall and generate specific scientific terminology.
- Bird images yield better VLM performance: VLMs generally perform better on bird images compared to fish and butterflies. This might be because bird datasets (like CUB) are more common in machine learning literature and often feature natural backgrounds, potentially leading to better pre-training alignment.
- Trait identification is more feasible than localization: VLMs are better at identifying the presence of a trait than at precisely locating it within an image (trait grounding and referring). This suggests a disconnect between their knowledge of biological concepts and their ability to visually ground that knowledge.
- Counting is a challenge for VLMs: Similar to other object counting tasks, VLMs exhibit difficulty in accurately counting biological traits.
- Prompting and model size matter: Larger models like GPT-4V and GPT-4o show improved reasoning capabilities with advanced prompting techniques (e.g., Chain-of-Thought), while smaller models benefit less.
- Fine-tuning is crucial: A biologically fine-tuned model (BioCLIP) significantly outperforms general VLMs on species classification, highlighting the importance of domain-specific training data.
Key Results
- Species Classification: Zero-shot accuracy on open-ended questions was notably weak across all VLMs (e.g., < 4% for top models), while MC questions showed better performance (up to 82.58% for GPT-4V on Bird-10K).
- Trait Identification: Performance was generally better than species classification, with GPT-4V achieving 82.18% on Fish-10K.
- Trait Grounding & Referring: A significant drop in accuracy was observed compared to trait identification, indicating struggles with visual localization.
- Prompting Techniques: GPT-4V and GPT-4o showed the most improvement with Chain-of-Thought prompting.
- Reasoning Hallucination Tests: GPT-4V and GPT-4o demonstrated the highest accuracy and lowest false confidence.
Limitations and Future Work
Limitations: * Limited Organism Scope: The dataset currently focuses on only three taxonomic groups (fish, birds, butterflies) due to resource constraints. * Potential Label Noise: While efforts were made, some label noise might be present in the dataset. * Proprietary Model Exclusion: Due to API costs, some proprietary VLMs (e.g., Gemini, Claude) were not evaluated.
Future Work: * Expand Organism Coverage: Include more diverse organisms to create a more comprehensive benchmark. * Fine-tuning VLMs: Explore the performance gains from fine-tuning VLMs on the VLM4Bio dataset. * Investigate RAG and Knowledge-Infused Prompting: Explore advanced techniques that leverage external knowledge bases. * Address Label Noise: Further refine the dataset to mitigate potential label noise.
Summary updated