Enhancing Vision-Language Models with Morphological and Taxonomic Knowledge: Towards Coral Recognition for Ocean Health

Hongyong Han, Wei Wang, Gaowei Zhang, Mingjie Li, Yi WangAAAI 2025 · DOI

Abstract

Automated coral reef monitoring is critical for environmental preservation, yet general-purpose Vision-Language Models (VLMs) struggle with the subtle morphological and taxonomic distinctions required for accurate coral identification. This paper introduces CORAL-Adapter, a parameter-efficient fine-tuning method that incorporates domain-specific biological knowledge into CLIP. By leveraging a new hierarchical dataset, HSCR16K, the proposed method achieves significant gains in few-shot and zero-shot coral recognition across diverse oceanic regions.

Introduction

Coral reefs support immense biodiversity but face unprecedented threats from climate change and pollution. Large-scale monitoring via automated image recognition is essential for conservation, but existing models face two major hurdles: a lack of specialized, taxonomically consistent data and the "look-alike" challenge. This challenge arises because many coral species exhibit subtle morphological differences (texture, structure) that generic CLIP prompts cannot capture, while some species look nearly identical despite belonging to entirely different taxonomic groups.

The CORAL-Adapter Framework

To address these challenges, the authors propose CORAL-Adapter, a plug-and-play module designed to inject expert knowledge into the CLIP vision-language model without requiring full fine-tuning.

Architecture of the CORAL-Adapter, showing the integration of morphological and taxonomic modules with the CLIP backbone.

The framework consists of two primary components:

  1. Morphological Adapter: Corals are often distinguished by repetitive texture and skeletal patterns. This adapter captures "second-order relationships" by computing the outer product of feature vectors from CLIP's visual encoder, effectively modeling complex textures better than standard linear layers.
  2. Taxonomic Adapter: This module aligns visual features with the biological "tree of life" (Family, Genus, Species). It uses a bottleneck structure to provide evolutionary context, helping the model distinguish between morphologically similar but taxonomically distinct species.

To ensure these adapters learn complementary information rather than redundant features, the model is optimized using a reversed Jensen-Shannon (JS) divergence loss alongside standard cross-entropy and similarity losses.

The HSCR16K Dataset

The researchers introduced the Hierarchical Stony Coral Recognition (HSCR16K) dataset to provide the necessary depth for training specialized models.

Taxonomic hierarchy of the HSCR16K dataset, illustrating the diversity and visual complexity of stony coral species.

HSCR16K contains 16,659 images covering 16 stony coral species across 8 families. Unlike previous benchmarks, it includes full taxonomic hierarchies and expert-crafted morphological descriptions detailing texture, color, and structure. The data spans 13 marine regions across the Pacific and Atlantic Oceans, ensuring robustness to varying underwater lighting and salinity conditions.

Key Results and Insights

CORAL-Adapter demonstrates superior performance across several challenging scenarios:

  • Few-Shot Learning: In a 1-shot setting (only one training image per class), the model achieved 41.4% accuracy, a massive 27.9% improvement over zero-shot CLIP.
  • Zero-Shot Generalization: The model outperformed existing adaptation methods by 10.8% when identifying bleaching-prone species not seen during training—a critical capability for early-warning environmental systems.
  • Cross-Ocean Robustness: When trained on Pacific corals and tested on Atlantic species, CORAL-Adapter maintained a 19.8% accuracy lead over competitors, proving its ability to generalize across different oceanic environments.

Conclusion

By integrating taxonomic hierarchy and morphological patterns, CORAL-Adapter transforms a general VLM into a specialized tool for marine biology. While currently dependent on expert descriptions, future iterations may leverage Large Language Models (LLMs) to scale these descriptions from scientific literature, paving the way for real-time coral monitoring on autonomous underwater vehicles (AUVs).

Summary updated

BibTeX
@inproceedings{han2025enhancing,
  author = {Hongyong Han and Wei Wang and Gaowei Zhang and Mingjie Li and Yi Wang},
  title = {{Enhancing Vision-Language Models with Morphological and Taxonomic Knowledge: Towards Coral Recognition for Ocean Health}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35023},
}
Figure