Benchmarking Large Language Models for Image Classification of Marine Mammals

Yijiashun Qi, Shuzhang Cai, Zunduo Zhao, Jiaming Li, Yanbin Lin, Zhiqiang Wang

Introduction

Marine mammals serve as critical indicators of ocean health, yet they face mounting threats from climate change and anthropogenic activities. While artificial intelligence has transformed image recognition, existing benchmarks predominantly focus on terrestrial animals or generic objects. This creates a significant "data gap" in marine biology, hindering the evaluation of state-of-the-art models—particularly multimodal Large Language Models (LLMs)—in specialized aquatic contexts. Identifying species like the "Baikal Seal" versus the "Steller Sea Lion" requires discerning subtle morphological traits in challenging environments characterized by low contrast, bubbles, or partial submergence. This study establishes a benchmark using a new specialized dataset to assess the readiness of AI for marine conservation.

Methods

The research evaluated four distinct classification strategies:

  1. Traditional Machine Learning (ML): Utilizing DINOv2 for image embedding extraction, followed by classification via K-Nearest Neighbors (KNN) and Support Vector Machines (SVM).
  2. Fine-tuned Deep Learning (DL): Standard architectures (e.g., ResNet, VGG) fine-tuned on the marine mammal dataset.
  3. Zero-shot Multimodal LLMs: Models such as GPT-4o, Gemini-1.5 Pro, and LLaVA were tested without domain-specific training. Prompting strategies included direct identification (0s) and Chain-of-Thought (0s-CoT), comparing open-ended queries against name-listed options.
  4. Multi-Agent System (MAS): A novel ensemble approach where multiple LLMs provide predictions to a "Critic Agent" that synthesizes the final result.
The Multi-Agent System (MAS) architecture uses a Critic Agent to synthesize predictions from multiple foundation models.

The Marine Mammal Dataset

The authors introduced a curated dataset of 1,423 images sourced from platforms like Flickr and GBIF. The dataset captures animals in diverse settings, including underwater, aerial, and boat-side perspectives. Crucially, the data is organized into a three-level hierarchy to test different granularities of classification.

The dataset is organized into a taxonomic hierarchy covering four major groups and 65 specific species.

The hierarchy includes:

  • 4 Group-level classes: (e.g., Cetacean, Pinniped).
  • 10 Medium-level classes: (e.g., Dolphin, Whale, Seal).
  • 65 Species-level classes: (e.g., Orca, Narwhal, Baikal Seal).

Key Insights and Results

The benchmarking process revealed several critical findings regarding the performance of modern AI in specialized domains:

  • Traditional ML Superiority in Fine-Grained Tasks: Surprisingly, a traditional SVM using DINOv2 embeddings achieved the highest species-level accuracy at 64.62%, outperforming both GPT-4o and Gemini. This suggests that for highly specialized tasks, domain-specific feature extraction remains superior to general-purpose zero-shot LLMs.
  • The "Name-Listed" Advantage: LLM performance improved significantly when provided with a list of possible species. This constraint reduces "cognitive load" and mitigates hallucinations of non-existent species.
  • LLM Bias and Overthinking: LLMs exhibited a bias toward common species found in their training data (e.g., Bottlenose Dolphins). Furthermore, Chain-of-Thought (CoT) prompting unexpectedly decreased species-level accuracy for some models, suggesting that complex reasoning can sometimes lead models away from correct visual cues in zero-shot scenarios.
  • Group-Level Success: At the broadest level (e.g., distinguishing a whale from a seal), almost all models, including LLMs, achieved near 100% accuracy.

Conclusion

While modern LLMs and Multi-Agent Systems excel at broad category identification, traditional machine learning models still hold the advantage for fine-grained, species-level classification in marine biology. Future work will focus on expanding the dataset to include rarer species and developing MAS architectures that can integrate external biological databases to improve specialized reasoning.

Summary updated

BibTeX
@inproceedings{qi2024benchmarking,
  author = {Yijiashun Qi and Shuzhang Cai and Zunduo Zhao and Jiaming Li and Yanbin Lin and Zhiqiang Wang},
  title = {{Benchmarking Large Language Models for Image Classification of Marine Mammals}},
  booktitle = {ICKG},
  year = {2024},
  doi = {10.1109/ickg63256.2024.00040},
}
Figure