FishNet: A Large-scale Dataset and Benchmark for Fish Recognition, Detection, and Functional Trait Prediction

Faizan Farooq Khan, Xiang Li, Andrew J. Temple, Mohamed ElhoseinyICCV 2023 · OpenReview

Abstract

Aquatic species are essential components of the world’s ecosystem, and the preservation of aquatic biodiversity is crucial for maintaining proper ecosystem functioning. Unfortunately, increasing anthropogenic pressures such as overfishing, climate change, and coastal development pose significant threats to aquatic biodiversity. To address this challenge, it is necessary to design automatic aquatic species monitoring systems that can help researchers and policymakers better understand changes in aquatic ecosystems and take appropriate actions to preserve biodiversity. However, the development of such systems is impeded by a lack of large-scale diverse aquatic species datasets. To address the need for generalized systems that can recognize, locate, and predict a wide array of species and their functional traits, we present FishNet, a large-scale diverse dataset containing 94,532 meticulously organized images from 17,357 aquatic species, organized according to aquatic biological taxonomy (order, family, genus, and species). We further build three benchmarks—fish classification, fish detection, and functional trait prediction—to facilitate the development of aquatic species recognition systems and promote further research in the field of aquatic ecology.

Introduction

Monitoring aquatic biodiversity is a critical yet bottlenecked process for maintaining ecosystem health. Identifying species in the wild typically requires high-level expert knowledge and is incredibly time-consuming. While deep learning offers a potential solution, existing fish datasets are often too small, geographically limited, or focused on a tiny fraction of the world's >35,000 known fish species. Furthermore, ecologists require more than just taxonomic identification; they need data on "functional traits"—a species' position in the food web, habitat preferences, and movement patterns. Before FishNet, no large-scale benchmark combined visual data with these vital ecological attributes.

The FishNet Dataset

FishNet stands as the most diverse fish dataset to date, covering 94,532 images across 17,357 species—nearly half of all known fish species on Earth. For context, previous major datasets like WildFish++ covered only ~2,300 species.

Overview of the FishNet taxonomic hierarchy and functional trait integration.

The dataset includes 114,375 object instances with bounding boxes and 22 functional traits, such as trophic level, feeding pathway, salinity preference, and nutrient content (e.g., Omega-3 or Iron).

Methodology

Researchers from KAUST built FishNet by integrating data from FishBase and iNaturalist, focusing on three pillars:

  1. Taxonomic Rigor: The dataset is organized into 8 classes, 83 orders, 463 families, and 3,826 genera.
  2. Quality Control: Thousands of images were manually filtered to remove non-photographic noise such as sketches, coins, stamps, and larval fish. For detection, Amazon Mechanical Turk (AMT) was used with a strict consensus rule: three independent annotators had to agree on bounding boxes with an Intersection over Union (IoU) of >0.9.
  3. Benchmark Modeling: The team evaluated the dataset using state-of-the-art architectures, including ResNets, Vision Transformers (ViT), and ConvNeXt, across three tasks: classification, detection, and trait prediction.

Key Insights

The "Long-Tail" Reality

FishNet reflects the natural world, where a few species are common and thousands are rare. This makes it a premier benchmark for "long-tail" learning, challenging models to recognize categories from very few examples.

Traits over Taxa

The authors argue that predicting functional traits is often more useful for conservation than species names. An AI identifying an unknown fish as a "high-trophic level predator in a coral reef" provides immediate actionable data even if the exact species is unclear.

Zero-Shot Ecology

A significant finding is that models can predict the traits of species they have never seen before (zero-shot) with reasonable accuracy (~80% for habitat traits), suggesting that visual features are strongly correlated with ecological roles across the tree of life.

Key Results

The best-performing model (ConvNeXt with class-balanced loss) achieved 61.38% accuracy at the family level and 68.81% at the order level. Using the TOOD (Task-aligned One-stage Object Detection) framework, the researchers achieved an $mAP_{50}$ of 56.5 at the order level.

Qualitative results of fish classification and detection across different families.

Regarding functional traits, the model predicted individual habitat traits (like "Freshwater" vs "Saltwater") with over 90% accuracy. Trophic level (position in the food chain) was predicted with a Mean Absolute Error (MAE) of 0.45.

Limitations and Future Work

Performance drops significantly for rare species in the "long tail," and identifying a species from just one or two images remains a major hurdle. Additionally, environmental noise such as low contrast and turbid water continues to challenge detection models. Future work may involve using FishNet to study climate change impacts by monitoring shifts in species distributions and functional roles over time.

Reference: Khan, F. F., Li, X., Temple, A. J., & Elhoseiny, M. (2023). FishNet: A Large-scale Dataset and Benchmark for Fish Recognition, Detection, and Functional Trait Prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

Summary updated

Abstract

Aquatic species are essential components of the world’s ecosystem, and the preservation of aquatic biodiversity is crucial for maintaining proper ecosystem functioning. Unfortunately, increasing anthropogenic pressures such as overfishing, climate change, and coastal development pose significant threats to aquatic biodiversity. To address this challenge, it is necessary to design an automatic aquatic species monitoring systems that can help researchers and policymakers better understand changes in aquatic ecosystems and take appropriate actions to preserve biodiversity. However, the development of such systems is impeded by a lack of large-scale diverse aquatic species datasets. Existing aquatic species recognition datasets generally have a limited number of species, nor do they provide functional trait data, and so have only narrow potential for application. To address the need for generalized systems that can recognize, locate, and predict a wide array of species and their functional traits, we present FishNet, a large-scale diverse dataset containing 94,532 meticulously organized images from 17,357 aquatic species, organized according to aquatic biological taxonomy (order, family, genus, and species). We further build three benchmarks, i.e., fish classification, fish detection, and functional trait prediction, inspired by ecological research needs, to facilitate the development of aquatic species recognition systems, and promote further research in the field of aquatic ecology. Our FishNet dataset has the potential to encourage the development of more accurate and effective tools for the monitoring and protection of aquatic ecosystems, and hence take effective action toward the conservation of our planet’s aquatic biodiversity. Our dataset and code will be released at https://fishnet-2023.github.io/.
BibTeX
@inproceedings{khan2023fishnet,
  author = {Faizan Farooq Khan and Xiang Li and Andrew J. Temple and Mohamed Elhoseiny},
  title = {{FishNet: A Large-scale Dataset and Benchmark for Fish Recognition, Detection, and Functional Trait Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=wbGWZXhtj3},
}
Figure