VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images

M. Maruf, Arka Daw, Kazi Sajeed Mehrab, Harish Babu Manogaran, Abhilash Neog, Medha Sawhney, Mridul Khurana, James P. Balhoff, Yasin Bakis, Bahadir Altintas, Matthew J. Thompson, Elizabeth G. Campolongo, Josef C. Uyeda, Hilmar Lapp, Henry L. Bart Jr., Paula M. Mabee, Yu Su, Wei-Lun Chao, Charles V. Stewart, Tanya Y. Berger-Wolf, Wasila M. Dahdul, Anuj Karpatne

TL;DR

This paper introduces VLM4Bio, a new benchmark dataset designed to evaluate the capabilities of large vision-language models (VLMs) in organismal biology. They found that while VLMs show promise, they struggle with complex biological reasoning and localization tasks, highlighting a significant gap between current VLM performance and the needs of scientific discovery in this field.

What Problem Does This Paper Address?

The explosion of biodiversity data, particularly in the form of images, presents a massive opportunity for scientific discovery in organismal biology. Traditionally, extracting biological traits from these images has been a labor-intensive process requiring expert knowledge. The advent of powerful Vision-Language Models (VLMs) offers a potential solution to automate this process. However, existing VLMs are often trained on general datasets and may lack the domain-specific knowledge and reasoning abilities required for biological tasks. This paper addresses the critical need for a specialized benchmark to evaluate how well current VLMs can understand and reason about biological images, specifically for trait discovery, and to identify their limitations in this domain.

Methods

The authors developed the VLM4Bio dataset, comprising approximately 469,000 question-answer pairs derived from 30,000 images of fishes, birds, and butterflies. They designed five biologically relevant tasks: species classification, trait identification, trait grounding (locating a trait), trait referring (identifying a trait from a location), and trait counting. To evaluate VLMs, they employed a zero-shot approach, meaning the models were tested without any specific fine-tuning on the VLM4Bio dataset. They also investigated the impact of different prompting techniques (contextual, dense captioning, and chain-of-thought) and reasoning hallucination tests (False Confidence Test and None of the Above Test) on VLM performance.

Datasets

The primary contribution is the VLM4Bio dataset, which includes:

  • Images: Approximately 30,000 images of fishes (Fish-10K), birds (Bird-10K), and butterflies (Butterfly-10K).
  • Question-Answer Pairs: Around 469,000 pairs covering five tasks.
  • Existing Datasets Used:
    • FishAIR [24] and Integrated Digitized Biocollections (iDigBio) [26] for fish images.
  • Jiggins Heliconius Collection dataset [35] for butterfly images.
  • CUB-200-2011 [61] for bird images.

VLM4Bio differs from existing benchmarks by focusing specifically on organismal biology and offering a diverse set of biologically relevant tasks that require both predictive accuracy and visual reasoning.

Key Insights and Ideas

  • VLMs struggle with open-ended biological questions: While VLMs perform better on multiple-choice questions, their accuracy drops significantly when asked to provide answers without options, indicating a gap in their ability to recall and generate specific scientific terminology.
  • Bird images yield better VLM performance: VLMs generally perform better on bird images compared to fish and butterflies. This might be because bird datasets (like CUB) are more common in machine learning literature and often feature natural backgrounds, potentially leading to better pre-training alignment.
  • Trait identification is more feasible than localization: VLMs are better at identifying the presence of a trait than at precisely locating it within an image (trait grounding and referring). This suggests a disconnect between their knowledge of biological concepts and their ability to visually ground that knowledge.
  • Counting is a challenge for VLMs: Similar to other object counting tasks, VLMs exhibit difficulty in accurately counting biological traits.
  • Prompting and model size matter: Larger models like GPT-4V and GPT-4o show improved reasoning capabilities with advanced prompting techniques (e.g., Chain-of-Thought), while smaller models benefit less.
  • Fine-tuning is crucial: A biologically fine-tuned model (BioCLIP) significantly outperforms general VLMs on species classification, highlighting the importance of domain-specific training data.

Key Results

  • Species Classification: Zero-shot accuracy on open-ended questions was notably weak across all VLMs (e.g., < 4% for top models), while MC questions showed better performance (up to 82.58% for GPT-4V on Bird-10K).
  • Trait Identification: Performance was generally better than species classification, with GPT-4V achieving 82.18% on Fish-10K.
  • Trait Grounding & Referring: A significant drop in accuracy was observed compared to trait identification, indicating struggles with visual localization.
  • Prompting Techniques: GPT-4V and GPT-4o showed the most improvement with Chain-of-Thought prompting.
  • Reasoning Hallucination Tests: GPT-4V and GPT-4o demonstrated the highest accuracy and lowest false confidence.

Limitations and Future Work

Limitations: * Limited Organism Scope: The dataset currently focuses on only three taxonomic groups (fish, birds, butterflies) due to resource constraints. * Potential Label Noise: While efforts were made, some label noise might be present in the dataset. * Proprietary Model Exclusion: Due to API costs, some proprietary VLMs (e.g., Gemini, Claude) were not evaluated.

Future Work: * Expand Organism Coverage: Include more diverse organisms to create a more comprehensive benchmark. * Fine-tuning VLMs: Explore the performance gains from fine-tuning VLMs on the VLM4Bio dataset. * Investigate RAG and Knowledge-Infused Prompting: Explore advanced techniques that leverage external knowledge bases. * Address Label Noise: Further refine the dataset to mitigate potential label noise.

Summary updated

Abstract

Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images. The code and datasets for running all the analyses reported in this paper can be found at https://github.com/sammarfy/VLM4Bio.
BibTeX
@inproceedings{maruf2024vlm,
  author = {M. Maruf and Arka Daw and Kazi Sajeed Mehrab and Harish Babu Manogaran and Abhilash Neog and Medha Sawhney and Mridul Khurana and James P. Balhoff and Yasin Bakis and Bahadir Altintas and Matthew J. Thompson and Elizabeth G. Campolongo and Josef C. Uyeda and Hilmar Lapp and Henry L. Bart Jr. and Paula M. Mabee and Yu Su and Wei-Lun Chao and Charles V. Stewart and Tanya Y. Berger-Wolf and Wasila M. Dahdul and Anuj Karpatne},
  title = {{VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure