Intelligent Fish Detection System with Similarity-Aware Transformer

Shengchen Li, Haobo Zuo, Changhong Fu, Zhiyong Wang, Zhiqiang Xu

TL;DR

This paper introduces FishViT, a lightweight, edge-deployed vision Transformer designed to automate fish detection during high-speed water-land transfer. By combining linear attention with a novel soft-thresholding denoising mechanism, FishViT achieves state-of-the-art accuracy ($AP_{50}$ of 94.7%) and runs at over 80 FPS on edge hardware, successfully isolating individual fish within dense, visually identical groups.

What Problem Does This Paper Address?

In industrial fisheries, the "water-land transfer" phase—moving live fish from culture vessels to transport vehicles via pipelines—is critical. To maintain fish health and market value, this process must be fast and minimize injury. Currently, monitoring and counting fish during this transfer relies heavily on manual labor, which is expensive, inefficient, and error-prone due to human fatigue.

Automating this task with computer vision is notoriously difficult. Fish slide down pipelines at high speeds, crowd together closely (causing severe occlusion), and look virtually identical. Furthermore, the environment is chaotic, filled with background noise like rushing water, splashing spume, and unpredictable lighting reflections. Standard Convolutional Neural Networks (CNNs) lack the global context needed to distinguish overlapping, identical fish, while classical Vision Transformers (ViTs) are too computationally heavy to run in real-time on edge devices.

Methods

The authors developed a complete edge intelligent vision system consisting of a physical pipeline setup (equipped with high-speed cameras and specialized bar lighting) and a novel deep learning architecture called FishViT.

Overview of the FishViT architecture

The FishViT architecture introduces three key innovations to achieve high speed and accuracy:

  1. Linear (Factorized) Attention: Standard self-attention has a computational complexity of $O(N^2)$, where $N$ is the number of pixels. FishViT uses a factorized attention mechanism that alters the order of matrix multiplications, dropping the complexity to $O(N)$, making it viable for real-time edge deployment.
  2. Soft-Threshold Attention (STAttention): To combat environmental noise, the authors integrated a soft-thresholding mechanism directly into the attention layer. Inspired by deep residual shrinkage networks, this module dynamically calculates a noise threshold for each feature channel. Elements below the threshold (like water splashes or glare) are zeroed out, effectively sharpening the edges of individual fish.
  3. Similarity-Aware Multi-Level Encoder: This parallel multi-branch structure processes features across multiple scales simultaneously. This ensures the model can accurately capture both small and large fish, preventing dense clusters of fish from being mistakenly grouped as a single object.
Detailed workflow of the STAttention module

Datasets

Because no public dataset existed for this specific industrial application, the authors collected and annotated a new benchmark:

  • 85 high-quality video sequences captured at 90 frames per second (FPS) from real-world water-land transfer scenarios.
  • The dataset features two primary commercial species: Black carp and Silver carp.
  • It includes annotations for six challenging environmental attributes: spume, overexposure, tumbling, dense, flowing, and underexposure.
  • The data is split into 50 sequences for training, 5 for validation, and 30 for testing.

Key Insights and Ideas

  • In-Network Denoising: Instead of treating image denoising as a separate pre-processing step, embedding soft-thresholding directly into the attention mechanism allows the model to use global context to decide what constitutes "noise" dynamically.
  • Overcoming the Transformer Speed Barrier: The paper demonstrates that by pairing factorized attention with an efficient Pooling Positional Encoding (PPE) scheme, Transformers can comfortably outperform CNNs on edge devices without sacrificing their signature global modeling capabilities.
  • Boundary Clarification: In environments where objects share identical textures and colors, multi-scale parallel encoding is vital. It allows the model to cross-reference fine edge details with broader semantic shapes to draw distinct boundaries around highly clustered targets.
Confidence maps showing how FishViT suppresses background noise

Key Results

Evaluated against state-of-the-art real-time detectors (including YOLOv8 and RT-DETR), FishViT demonstrated superior performance across the board:

  • Speed: Achieved an average inference speed of 82.3 FPS on an NVIDIA RTX 3060, fully satisfying real-time industrial requirements.
  • Accuracy: Reached a top $AP_{50}$ score of 94.7%, outperforming the massive YOLOv8-X (94.4%) while using a fraction of the parameters (18M vs. 165M).
  • Error Reduction: Drastically minimized classification errors ($E_{cls} = 0.11$), which is crucial for commercial sorting where misidentification directly impacts revenue.
  • Ablation Validation: Component tests showed that adding the STAttention module provided a +3.5% boost in $AP_{50}$, while the multi-level parallel encoder added +4.1%, proving the value of both architectural choices.

Limitations and Future Work

While FishViT performs exceptionally well, its current evaluation is limited to two freshwater fish species (Black carp and Silver carp) in a controlled pipeline environment. Its robustness across a wider variety of species with diverse body shapes or in open-water marine environments remains unverified.

The authors note that this detection system serves as a foundational component. Future work will focus on expanding the system to handle downstream tasks such as multi-object tracking, automated counting, and precise biomass/size estimation.

Summary updated

Abstract

Fish detection in water-land transfer has significantly contributed to the fishery. However, manual fish detection in crowd-collaboration performs inefficiently and expensively, involving insufficient accuracy. To further enhance the water-land transfer efficiency, improve detection accuracy, and reduce labor costs, this work designs a new type of lightweight and plug-and-play edge intelligent vision system to automatically conduct fast fish detection with high-speed camera. Moreover, a novel similarity-aware vision Transformer for fast fish detection (FishViT) is proposed to onboard identify every single fish in a dense and similar group. Specifically, a novel similarity-aware multi-level encoder is developed to enhance multi-scale features in parallel, thereby yielding discriminative representations for varying-size fish. Additionally, a new soft-threshold attention mechanism is introduced, which not only effectively eliminates background noise from images but also accurately recognizes both the edge details and overall features of different similar fish. 85 challenging video sequences with high framerate and high-resolution are collected to establish a benchmark from real fish water-land transfer scenarios. Exhaustive evaluation conducted with this challenging benchmark has proved the robustness and effectiveness of FishViT with over 80 FPS. Real work scenario tests validate the practicality of the proposed method. The code and demo video are available at https://github.com/vision4robotics/FishViT.
BibTeX
@misc{li2024intelligent,
  author = {Shengchen Li and Haobo Zuo and Changhong Fu and Zhiyong Wang and Zhiqiang Xu},
  title = {{Intelligent Fish Detection System with Similarity-Aware Transformer}},
  year = {2024},
  eprint = {2409.19323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.19323},
}

Loading related content...

Figure