A Marine Organism Detection Framework Based on Dataset Augmentation and CNN-ViT Fusion

Jiang Xiao, Yaxin Zhang, Mian Pan, Shuaishuai Lv, Gang Yang, Li Zhu, Jingbiao Liu, Haibin YuJournal of Marine Science and Engineering 2023 · DOI

TL;DR

This paper introduces a novel framework for underwater marine organism detection that tackles the challenges of small object sizes and imbalanced datasets. By combining advanced data augmentation techniques with a CNN-ViT fusion model built on YOLOv5, they achieve a significant 27% improvement in mean average precision (mAP) while maintaining real-time performance for deployment on underwater vehicles.

What Problem Does This Paper Address?

Underwater vision-based detection is crucial for marine exploration, ecological monitoring, and resource management. However, it faces unique challenges:

  • Image Degradation: Underwater environments suffer from light absorption and scattering, leading to poor color contrast and fog.
  • Small and Clustered Objects: Marine organisms can appear very small in images, and their clustered nature makes individual detection difficult.
  • Data Imbalance: Obtaining a diverse and balanced dataset of underwater organisms is challenging, with some species being much rarer than others.
  • Resource Constraints: Underwater platforms like ROVs and AUVs have limited computational power and battery life, requiring efficient models.

These issues collectively make accurate and real-time detection of marine life a significant hurdle.

Methods

The proposed framework integrates several key components:

  • Data Augmentation Strategy:
    • Random Expansion of Small Objects: This technique artificially increases the size of small targets within existing images without adding new data, making them easier for the model to detect.
  • Non-Overlapping Filling of Scarce Samples: This method addresses data imbalance by strategically inserting images of rare organisms into existing datasets without overlap, effectively boosting the representation of underrepresented species.
  • CNN-ViT Fusion Backbone:
    • YOLOv5 Baseline: The framework uses YOLOv5 as its foundation due to its efficiency.
  • Vision Transformer (ViT) Integration: ViT is incorporated into the backbone to leverage its ability to capture long-range dependencies and global context, complementing CNN's local feature extraction capabilities.
  • Deformable Convolution: This technique allows the convolution kernels to adapt their sampling locations, improving the model's ability to handle targets with varying shapes and scales, which is common in marine environments.
  • Trident Block: This module uses multiple dilated convolutions to create a multi-scale receptive field, further enhancing the extraction of features from organisms of different sizes.
  • Loss Function:
    • Focal Loss: Used for classification and object confidence, it helps the model focus on difficult-to-classify samples and rare positive samples.
  • CIoU Loss: Employed for bounding box regression, it provides a more accurate measure of overlap and distance between predicted and ground truth boxes, especially in dense detection scenarios.

Datasets

The primary dataset used in this study is the URPC2019 dataset. This dataset was collected by underwater robots equipped with cameras for the "Underwater Robot Picking Contest." It contains 5543 marked images and 1400 unmarked images for testing, featuring four target classes: echinus, holothurian, scallop, and starfish. The authors note that this dataset exhibits significant issues with small object sizes and class imbalance.

Key Insights and Ideas

  • Data Augmentation is Crucial for Underwater Tasks: The paper strongly emphasizes that standard data augmentation techniques are insufficient for underwater imagery. Novel methods like random expansion and non-overlapping filling are essential to overcome the specific challenges of small, rare, and imbalanced marine organism datasets.
  • CNN-ViT Fusion Offers Synergistic Benefits: Combining CNNs (for local features) with ViTs (for global context) proves more effective than either alone for capturing the diverse characteristics of marine organisms. This fusion allows for a more comprehensive understanding of the visual scene.
  • Adaptive Kernels (Deformable Convolution) are Key for Irregular Shapes: Marine organisms often have non-standard shapes (e.g., flat or slender). Deformable convolution's ability to adapt its receptive field is vital for accurately detecting these irregular forms, which fixed kernels struggle with.
  • Multi-Scale Feature Extraction is Essential: The trident block, with its varied dilation rates, highlights the importance of capturing features at multiple scales to handle the significant size variations of marine organisms.

Key Results

  • Significant mAP Improvement: The proposed framework achieved a 27% increase in mean average precision (mAP) compared to the baseline YOLOv5 model.
  • High Precision and Recall: Ablation studies demonstrated that the combined data augmentation and model improvements led to substantial gains in precision, recall, [email protected], and [email protected]:0.95. For instance, the proposed framework achieved a recall of 0.8853 and precision of 0.8917.
  • Real-Time Performance: The framework maintains a competitive frame rate, achieving 17.46 FPS on an embedded Jetson Xavier NX platform, making it suitable for deployment on autonomous underwater vehicles.
  • Improved Detection of Small and Rare Species: The framework showed marked improvements in detecting small scallops and rare holothurians, as evidenced by confusion matrices and qualitative results.

Limitations and Future Work

  • Model Size: While achieving real-time performance, the proposed framework has a slightly larger parameter count (64.2M) compared to some other models like YOLOv5 (47.0M).
  • Computational Cost of Augmentation: The data augmentation strategies, while effective, increase training time.
  • Future Work: The paper implicitly suggests that further optimization of the model architecture and augmentation strategies could lead to even better performance. The focus on embedded platforms also implies potential for further research into model compression and efficiency for deployment on more constrained hardware.

Summary updated

Abstract

Underwater vision-based detection plays an important role in marine resources exploration, marine ecological protection and other fields. Due to the restricted carrier movement and the clustering effect of some marine organisms, the size of some marine organisms in the underwater image is very small, and the samples in the dataset are very unbalanced, which aggravate the difficulty of vision detection of marine organisms. To solve these problems, this study proposes a marine organism detection framework with a dataset augmentation strategy and Convolutional Neural Networks (CNN)-Vision Transformer (ViT) fusion model. The proposed framework adopts two data augmentation methods, namely, random expansion of small objects and non-overlapping filling of scarce samples, to significantly improve the data quality of the dataset. At the same time, the framework takes YOLOv5 as the baseline model, introduces ViT, deformable convolution and trident block in the feature extraction network, and extracts richer features of marine organisms through multi-scale receptive fields with the help of the fusion of CNN and ViT. The experimental results show that, compared with various one-stage detection models, the mean average precision (mAP) of the proposed framework can be improved by 27%. At the same time, it gives consideration to both performance and real-time, so as to achieve high-precision real-time detection of the marine organisms on the underwater mobile platform.
BibTeX
@article{xiao2023marine,
  author = {Jiang Xiao and Yaxin Zhang and Mian Pan and Shuaishuai Lv and Gang Yang and Li Zhu and Jingbiao Liu and Haibin Yu},
  title = {{A Marine Organism Detection Framework Based on Dataset Augmentation and CNN-ViT Fusion}},
  journal = {Journal of Marine Science and Engineering},
  year = {2023},
  doi = {10.3390/jmse11040705},
}
Figure