A labeled data set of underwater images of fish and crab species from five mesohabitats in Puget Sound WA USA

Dara M. Farrell, Bridget E. Ferriss, Beth L. Sanderson, Karl Veggerby, Lauren Robinson, Anusua Trivedi, Shreyaan Pathak, Sreya Muppalla, Jane Wang, Dan Morris, Rahul DodhiaScientific 2023 · DOI

Abstract

The sustainable management of fisheries and aquaculture requires an understanding of how these activities interact with natural fish populations. GoPro cameras were used to collect an underwater video data set on and around shellfish aquaculture farms in an estuary in the NE Pacific from June to August 2017 and June to August 2018 to better understand habitat use by the local fish and crab communities. Images extracted from these videos were labeled to produce a data set that is suitable for use in training computer vision models. The labeled data set contains 77,739 images sampled from the collected video; 67,990 objects (fishes and crustaceans) have been annotated in 30,384 images (the remainder have been annotated as "empty"). The metadata of the data set also indicates whether a physical magenta filter was used during video collection to counteract reduced visibility. These data have the potential to help researchers address system-level and in-depth regional shellfish aquaculture questions related to ecosystem services and shellfish aquaculture interactions.

Introduction

Understanding the ecological interactions between human activities, such as shellfish aquaculture, and local marine populations is critical for sustainable management. However, monitoring these interactions in the field presents significant logistical challenges. Traditional sampling methods like beach seines often snag on aquaculture structures, while diver surveys are costly and may disturb the animals being studied.

Underwater video offers a non-invasive, cost-effective alternative, but the resulting "data deluge" makes manual analysis impractical. While computer vision (CV) can automate this process, most existing datasets are tailored to clear, tropical waters with artificial lighting. This paper introduces a large-scale, labeled dataset from the turbid, naturally lit, and ecologically complex waters of Puget Sound, Washington, designed to train robust CV models for real-world marine environments.

Figure 1: Map of Puget Sound sampling locations and representative photos of the five studied mesohabitats: flipbags (FB), oysters on bottom (OB), clams (CL), eelgrass (EG), and sediment (SD).

Methodology

Researchers from the University of Washington, NOAA, and Microsoft deployed GoPro cameras (Hero 3+ and 4) at nine farm locations across Puget Sound during the summers of 2017 and 2018.

Sampling and Hardware

To ensure a representative sample, cameras recorded for 2-minute intervals every 10 minutes during a 3-hour window around high tide. In 2018, physical magenta filters were added to mitigate the heavy green tint caused by high algae concentrations in the estuary, improving visibility for both human annotators and future AI models.

Labeling Pipeline

The dataset was produced through a rigorous human-in-the-loop pipeline involving over a dozen annotators. A two-pass system was employed: an initial annotation phase (drawing bounding boxes around fish and crabs) followed by a quality-check pass to resolve discrepancies. This process aimed for an accuracy rate exceeding 95%.

Dataset Overview

The resulting NOAA Puget Sound Nearshore Fish 2017–2018 dataset is hosted on Mendeley Data and LILA BC. It consists of 77,739 images at 1920 × 1080 resolution.

  • Annotated Content: 30,384 images contain 67,990 individual objects (fish and crustaceans).
  • Negative Samples: 47,355 images are labeled as "empty." These are critical for training models to distinguish between marine life and background "noise" like swaying eelgrass or silt.
  • Format: Annotations use the standard COCO (Common Objects in Context) JSON format, ensuring compatibility with modern machine learning frameworks.
Figure 2: Examples of image quality across the five mesohabitats, illustrating the range of visibility and common obstructions like bubbles.

Key Findings and Ecological Insights

The dataset captures a diverse range of species and habitats. "Oysters in flipbags" was the most frequently sampled habitat (28.1%), followed by "oysters on bottom" (17.9%). In images where animals were present, fish appeared in approximately 75% of instances, while crabs were present in 16%. Common species identified include surf perch, sculpins, flatfish, and Dungeness crabs.

A key strength of this dataset is its inclusion of real-world "noise"—images obscured by bubbles, silt, and algae. While challenging for AI, these conditions are representative of actual field deployments in temperate estuaries, ensuring that models trained on this data are "battle-hardened" for practical use.

Limitations and Future Work

The authors acknowledge several limitations. Crabs in "oysters on bottom" habitats are often camouflaged, leading to likely underestimation in the annotations. Additionally, because recordings were restricted to slack tide, highly mobile species that move with the tide may be underrepresented. Currently, labels are broad (fish vs. crab); future work could involve species-level identification to enable more granular ecological research.

This dataset marks a significant advancement toward "automated ecology," providing the necessary foundations for intelligent tools to monitor and protect marine ecosystems.

Summary updated

Abstract

The sustainable management of fisheries and aquaculture requires an understanding of how these activities interact with natural fish populations. GoPro cameras were used to collect an underwater video data set on and around shellfish aquaculture farms in an estuary in the NE Pacific from June to August 2017 and June to August 2018 to better understand habitat use by the local fish and crab communities. Images extracted from these videos were labeled to produce a data set that is suitable for use in training computer vision models. The labeled data set contains 77,739 images sampled from the collected video; 67,990 objects (fishes and crustaceans) have been annotated in 30,384 images (the remainder have been annotated as "empty"). The metadata of the data set also indicates whether a physical magenta filter was used during video collection to counteract reduced visibility. These data have the potential to help researchers address system-level and in-depth regional shellfish aquaculture questions related to ecosystem services and shellfish aquaculture interactions.
BibTeX
@inproceedings{farrell2023labeled,
  author = {Dara M. Farrell and Bridget E. Ferriss and Beth L. Sanderson and Karl Veggerby and Lauren Robinson and Anusua Trivedi and Shreyaan Pathak and Sreya Muppalla and Jane Wang and Dan Morris and Rahul Dodhia},
  title = {{A labeled data set of underwater images of fish and crab species from five mesohabitats in Puget Sound WA USA}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41597-023-02557-6},
}
Figure