Detecting and Counting Oysters

Behzad Sadrfaridpour, Yiannis Aloimonos, Miao Yu, Yang Tao, Donald Webster

Abstract

Oysters are an essential species in the Chesapeake Bay living ecosystem, acting as filter feeders that significantly improve water quality. Despite their importance, oyster populations have plummeted, necessitating restoration programs. This work presents an automated system for monitoring oysters using a Remotely Operated Vehicle (ROV) and deep learning. We constructed a database of oysters from the Chesapeake Bay, implemented Mask R-CNN for instance segmentation, and developed a tracking-by-detection pipeline to count oysters in video sequences. Our system achieves approximately 80% counting accuracy, offering a scalable alternative to manual monitoring.

Introduction

Oysters are often called the "vacuum cleaners" of the Chesapeake Bay. They provide critical ecosystem services by filtering water and creating habitats for other marine life. However, global oyster populations have declined sharply. To manage restoration efforts, scientists need accurate "reef metrics," such as density and size distribution.

Traditional monitoring involves divers manually counting small samples, a process that is slow, expensive, and difficult to scale. While sonar can map seafloor topography, it lacks the resolution to identify individual oysters. Optical imaging provides the necessary detail but faces significant challenges: high turbidity ("underwater snow"), low light, and the visual similarity between oysters and surrounding sediment-covered rocks or empty shells.

Methods

The Tracking-by-Detection Pipeline

To automate the counting process without double-counting individuals as the ROV moves, we implemented a tracking-by-detection pipeline.

The tracking-by-detection pipeline architecture, showing the interaction between the object detector, tracker, and matcher.

The pipeline consists of three main stages:

  1. Detection & Segmentation: We utilize Mask R-CNN to identify and segment individual oysters. Unlike standard detectors that only provide bounding boxes, Mask R-CNN generates pixel-level masks, which are essential for future work in calculating physical dimensions like area and volume.
  2. Tracking: We employ Kernelized Correlation Filters (KCF) for tracking. KCF is computationally efficient, making it suitable for potential real-time deployment on ROVs.
  3. Association: New detections are matched with existing trackers using Intersection over Union (IoU). A threshold of 0.2 is used to determine if a detection and a tracker represent the same object.

Mask R-CNN Architecture

The core of our detection system is the Mask R-CNN architecture, which extends Faster R-CNN by adding a branch for predicting segmentation masks.

The Mask R-CNN architecture used for oyster detection, featuring a CNN backbone, Region Proposal Network (RPN), and RoIAlign.

We experimented with various ResNet backbones (ResNet-50 and ResNet-101). Our findings indicated that the ResNet50-C4 backbone provided the best overall Average Precision (AP), while ResNet50-DC5 (using dilated convolutions) performed best at the AP50 metric, suggesting that high-resolution feature maps are critical for detecting small, textured objects in turbid environments.

Dataset and Challenges

Since no standard benchmark existed for oysters in turbid bay water, we created a custom dataset. Data was collected in the Chesapeake Bay using a BlueROV2 equipped with a GoPro HERO7 and external LED lighting. Due to extreme turbidity, oysters were only visible within three feet of the camera.

We annotated 1,290 individual oyster instances using a human-in-the-loop approach to accelerate labeling. The dataset was split into 827 instances for training, with the remainder reserved for validation and testing. A key heuristic used during ground-truthing was the "cup-side up" assumption: objects oriented cup-side up were classified as live oysters, as sediment-covered shells and rocks are otherwise nearly indistinguishable.

Results

The system demonstrated strong performance in challenging conditions:

  • Detection: The best-performing model achieved an AP50 of 81.8.
  • Counting: In video-based tests, the system reached an average counting accuracy of 79.8% compared to manual human counts.

The tracking system effectively mitigated double-counting caused by ROV movement. However, performance degradation was noted when oysters were heavily obscured by sediment or "fouled" by other organisms like bryozoa.

Conclusion and Future Work

This research demonstrates the feasibility of using deep learning and ROVs for automated oyster population monitoring. While the current system is highly effective, future work will focus on distinguishing between live oysters and empty shells ("dead-loss") to improve population estimates. Additionally, we aim to integrate this vision system into a fully autonomous underwater vehicle (AUV) capable of both monitoring and autonomous harvesting.

Summary updated

Abstract

Oysters are an essential species in the Chesapeake Bay living ecosystem. Oysters are filter feeders and considered the vacuum cleaners of the Chesapeake Bay that can considerably improve the Bay's water quality. Many oyster restoration programs have been initiated in the past decades and continued to date. Advancements in robotics and artificial intelligence have opened new opportunities for aquaculture. Drone-like ROVs with high maneuverability are getting more affordable and, if equipped with proper sensory devices, can monitor the oysters. This work presents our efforts for videography of the Chesapeake bay bottom using an ROV, constructing a database of oysters, implementing Mask R-CNN for detecting oysters, and counting their number in a video by tracking them.
BibTeX
@inproceedings{sadrfaridpour2021detecting,
  author = {Behzad Sadrfaridpour and Yiannis Aloimonos and Miao Yu and Yang Tao and Donald Webster},
  title = {{Detecting and Counting Oysters}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=KFtmUiZAkx},
}
Figure