Detecting and Counting Oysters

Behzad Sadrfaridpour, Yiannis Aloimonos, Miao Yu, Yang Tao, Donald Webster

TL;DR

Researchers at the University of Maryland have developed an automated system to detect and count oysters in the wild using underwater drones (ROVs) and deep learning. By combining Mask R-CNN for instance segmentation with Kernelized Correlation Filters (KCF) for tracking, the system can monitor oyster populations in the turbid waters of the Chesapeake Bay with roughly 80% counting accuracy.

What Problem Does This Paper Address?

Oysters are the "vacuum cleaners" of marine ecosystems, vital for filtering water and providing habitat. However, oyster populations have plummeted globally. To restore them, scientists need to monitor "reef metrics" like density and size distribution.

Currently, this is a grueling manual process: divers or researchers manually count small samples (often just 100 oysters per site), which is slow, expensive, and doesn't scale. While sonar can map the seafloor, it lacks the resolution to identify individual oysters. Optical imaging is a better alternative but faces massive hurdles: underwater "snow" (turbidity), low light, and the fact that oysters often look exactly like the sediment-covered rocks and empty shells surrounding them.

Methods

The authors implement a tracking-by-detection pipeline consisting of three main stages:

  1. Detection & Segmentation: They use Mask R-CNN, a state-of-the-art deep learning architecture. Unlike standard object detectors that just draw boxes, Mask R-CNN creates a pixel-perfect "mask" for each oyster. This is crucial because it allows for future calculations of physical dimensions (size/area). They tested several "backbones" (the underlying feature extractors), including various versions of ResNet-50 and ResNet-101.
  2. Tracking: To count oysters in a video without double-counting, the system must recognize the same oyster across consecutive frames. The authors chose the KCF (Kernelized Correlation Filters) tracker because it is computationally efficient—essential for potential real-time use on an ROV.
  3. Association: The system matches new detections with existing trackers using Intersection over Union (IoU). If a detected box overlaps significantly with a tracked box (threshold of 0.2), they are considered the same object. New trackers are spawned for unmatched detections, and old ones are deleted if they vanish for too many frames.

Datasets

The researchers built a custom dataset because no standard benchmark for "oysters in turbid bay water" existed:

  • Data Collection: Captured using a BlueROV2 equipped with a GoPro HERO7 and external LED lights in the Chesapeake Bay.
  • The Challenges: The water was so turbid that oysters were only visible within three feet of the camera.
  • Annotation: They labeled 1,290 individual oyster instances. To speed up the process, they used a "human-in-the-loop" approach: training a model on 50 images, using it to suggest labels for the rest, and then manually correcting those suggestions.
  • Split: 827 instances for training, with the remainder used for validation and testing.

Key Insights and Ideas

  • Segmentation vs. Detection: The authors emphasize that segmentation (masks) is superior to simple bounding boxes for aquaculture. Knowing the exact shape of the oyster is the first step toward building robots that can eventually harvest them autonomously.
  • The "Cup-Side Up" Assumption: A major challenge in ground-truthing is that live oysters and empty shells look nearly identical when covered in sediment. The authors assumed that any object oriented "cup-side up" was a live oyster, a necessary heuristic for underwater visual monitoring.
  • Backbone Matters: Interestingly, the ResNet50-C4 backbone provided the best overall Average Precision (AP), while ResNet50-DC5 (which uses dilated convolutions) was best at the more lenient AP50 metric. This suggests that higher-resolution feature maps are vital for detecting small, textured objects like oysters.

Key Results

  • Detection Accuracy: The best model achieved an AP50 of 81.8, meaning it was highly effective at identifying the presence of oysters even in difficult conditions.
  • Counting Accuracy: In video tests comparing the algorithm to manual human counts, the system achieved an average accuracy of 79.8%.
  • Robustness: The tracking system successfully mitigated double-counting caused by the ROV's movement, though performance dropped when oysters were heavily obscured by sediment or "fouled" by other marine organisms like bryozoa.

Limitations and Future Work

  • The "Empty Shell" Problem: The current model still struggles to distinguish between live oysters and empty shells (dead-loss), which can lead to overcounting the active population.
  • Turbidity Limits: The system is limited by physics; if the water is too murky to see past three feet, the ROV must fly very close to the bottom, which increases the risk of stirring up more sediment.
  • Automation: While the counting is automated, the ROV was controlled manually. Future work aims to integrate this vision system into a fully autonomous underwater vehicle (AUV) capable of both monitoring and harvesting.

The code for this project is available on GitHub.

Summary updated

Abstract

Oysters are an essential species in the Chesapeake Bay living ecosystem. Oysters are filter feeders and considered the vacuum cleaners of the Chesapeake Bay that can considerably improve the Bay's water quality. Many oyster restoration programs have been initiated in the past decades and continued to date. Advancements in robotics and artificial intelligence have opened new opportunities for aquaculture. Drone-like ROVs with high maneuverability are getting more affordable and, if equipped with proper sensory devices, can monitor the oysters. This work presents our efforts for videography of the Chesapeake bay bottom using an ROV, constructing a database of oysters, implementing Mask R-CNN for detecting oysters, and counting their number in a video by tracking them.
BibTeX
@inproceedings{sadrfaridpour2021detecting,
  author = {Behzad Sadrfaridpour and Yiannis Aloimonos and Miao Yu and Yang Tao and Donald Webster},
  title = {{Detecting and Counting Oysters}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=KFtmUiZAkx},
}

Loading related content...

Figure