Instance Segmentation of Benthic Scale Worms at a Hydrothermal Site

Bhuvan Malladihalli Shashidhara, Mitchell Scott, Aaron MarburgWACV 2020

TL;DR

This paper introduces a novel hybrid U-Net and VGG-16 CNN pipeline for instance segmentation of scale worms in challenging deep-sea hydrothermal vent imagery. The approach achieves a respectable 0.671 AP@[0.5] despite limited training data and the inherent difficulties of camouflaged subjects and dynamic environments, paving the way for automated ecological monitoring.

What Problem Does This Paper Address?

Deep-sea hydrothermal vents are vibrant ecosystems, but studying them is incredibly difficult due to extreme depths, pressures, and temperatures. Traditional methods like in-situ video surveys are time-consuming and challenging. While cabled observatories like the Ocean Observatories Initiative (OOI) Regional Cabled Array (RCA) now provide high-definition video, manually analyzing the vast archives of footage for ecological metrics is impractical. This paper focuses on automating the identification and segmentation of benthic scale worms, a common and mobile macrofaunal species, from this challenging imagery. The key challenges include:

  • Camouflage: Scale worms are often well-camouflaged against complex, dynamic backgrounds.
  • Environmental Distortions: Schlieren (waviness due to heat) from vent fluids can distort images.
  • Limited Labeled Data: Creating large, accurately labeled datasets for deep-sea imagery is expensive and difficult.
  • Dynamic Scenes: Imaging conditions change over time due to factors like equipment failure or biological growth, requiring frequent model retraining.

Methods

The authors propose a two-stage hybrid inference pipeline:

  1. U-Net for Semantic Segmentation: A U-Net architecture, known for its effectiveness in biomedical image segmentation with limited data, is used to generate initial segmentation masks. This network learns color and texture features. To improve robustness, batch normalization layers were added, and the network was trained using a binary cross-entropy loss with an Adam optimizer. The U-Net operates on smaller image patches (256x256 pixels) to handle the high-resolution input.
  2. VGG-16 CNN for Instance Verification: To reduce the high number of false positives generated by the U-Net, a VGG-16 CNN is employed as a binary classifier. This network takes the U-Net's predicted masks and further verifies them, outputting a probability score for whether a detected region is indeed a scale worm. This stage helps to filter out false positives and refine the segmentation.

The pipeline incorporates several pre- and post-processing steps:

  • Patch Extraction: Input images are split into overlapping patches.
  • Center-Crop Stitching: Patches are stitched back together to reconstruct a full-image mask, ensuring scale worms split across patch boundaries are captured.
  • Probability and Area Thresholding: Initial filtering of U-Net outputs.
  • Morphological Operations: Dilation is used to fill holes in masks.
  • Connected Component Analysis: To identify candidate regions.
  • VGG-16 Classification: Final validation of candidate regions.

Datasets

The primary dataset for this study was derived from CamHD video footage of the "Mushroom" hydrothermal vent at Axial Seamount.

  • Training Data:
    • S1 & S2: Manually labeled patch-masks from three specific regions of interest, containing 261 and 276 patches respectively.
  • RCP (Randomly Cropped Patches): 392 randomly cropped patches from "empty" Mushroom scenes, manually inspected to ensure they contained no scale worms.
  • CNP (Custom Negative Patches): 995 custom negative patches from locations with high false positive rates, specifically curated to help the VGG-16 distinguish between scale worms and their environment.
  • Test Set: A set of 45 images, densely labeled to include every visible scale worm, used for network testing.

The authors note that creating large, densely labeled test sets was expensive, influencing their experimental design.

Key Insights and Ideas

  • Hybrid Approach Effectiveness: Combining a semantic segmentation network (U-Net) with a classification network (VGG-16) significantly improves instance segmentation accuracy by addressing both detection and false positive reduction.
  • VGG-16 as a Bottleneck for Retraining: The VGG-16 CNN portion of the pipeline appears to be more sensitive to additional training data than the U-Net. This is a crucial insight for practical deployment, as it suggests that incremental retraining for dynamic scenes can be more efficiently focused on the VGG-16, simplifying maintenance.
  • Importance of Custom Negative Patches: The inclusion of CNP patches for VGG-16 training led to a noticeable improvement in performance, highlighting the value of carefully curated negative examples for difficult segmentation tasks.
  • Patch-wise Labeling Sufficiency: The pipeline's ability to work with sparse, patch-level labeling is a significant advantage for real-world applications where dense labeling is infeasible.

Key Results

  • The proposed hybrid U-Net + VGG-16 CNN pipeline achieved an Average Precision (AP) of 0.671 at an Intersection over Union (IoU) threshold of 0.5 (AP@[0.5]).
  • This performance was achieved with a relatively small training dataset.
  • The authors found that increasing the training data generally improved performance, with a notable jump when custom negative patches (CNP) were added to the VGG-16 training set.
  • The best performing configuration (U-V2 + VGG-V3) utilized a U-Net trained on both S1 and S2 datasets, and a VGG-16 trained on S1, S2, RCP, and CNP datasets.

Limitations and Future Work

  • False Negatives: The pipeline intentionally ignores large, green scale worms with unique appearances to simplify initial training. Additionally, smaller scale worms in unusual poses (e.g., on their side or back) were sometimes missed due to insufficient training data for these orientations.
  • Limited Test Set: The evaluation was performed on a relatively small test set due to the cost of creating large, densely labeled datasets.
  • Confidence Score Limitation: The confidence score for instance segmentation is solely derived from the VGG-16 model, and the paper does not quantify errors from the U-Net model directly.

Future work directions include:

  • Multi-Class Segmentation: Expanding the pipeline to identify other genera of macrofauna at Mushroom.
  • Transfer Learning: Investigating transfer-learning approaches, given the limited labeled data.
  • Generalizability: Applying the pipeline to other domains to assess the generalizability of the observations.
  • Ecological Statistics: Utilizing the segmentation model to extract spatial and temporal statistics for correlation with environmental data, enhancing the study of benthic marine life.

Summary updated

BibTeX
@inproceedings{shashidhara2020instance,
  author = {Bhuvan Malladihalli Shashidhara and Mitchell Scott and Aaron Marburg},
  title = {{Instance Segmentation of Benthic Scale Worms at a Hydrothermal Site}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure