FCFormer: fish density estimation and counting in recirculating aquaculture system

Kaijie Zhu, Xinting Yang, Caiwei Yang, Tingting Fu, Pingchuan Ma, Weichen HuFrontiers in Marine Science 2024 · DOI

TL;DR

Researchers have developed FCFormer, a Transformer-based deep learning framework designed to accurately count fish and estimate density in crowded aquaculture tanks. By combining global feature extraction with adaptive feature fusion, the model achieves over 97% accuracy, significantly outperforming traditional CNN-based methods in handling occlusion and small, distant targets.

What Problem Does This Paper Address?

In recirculating aquaculture systems (RAS), knowing exactly how many fish are in a tank is vital for managing feed, monitoring health, and predicting survival rates. However, counting fish is a nightmare for computer vision.

Traditional Convolutional Neural Networks (CNNs) often struggle in these environments for three main reasons:

  1. Mutual Occlusion: Fish swim over and under each other, hiding parts of their bodies.
  2. Rapid Movement: Fast-moving fish create motion blur.
  3. Scale Variation: Fish near the camera look huge, while those further away are tiny, blurry dots.

CNNs typically have a limited "receptive field," meaning they are great at seeing local textures but poor at understanding the global context of a crowded school. This paper introduces a Transformer-based approach to capture that missing global information.

Methods

The authors propose a framework called FCFormer (Fish Count Transformer), which moves away from simple object detection toward density map regression. Instead of drawing a box around every fish, the model predicts a map where the intensity of pixels represents fish density.

The architecture consists of three novel components:

  • Twins-SVT Backbone: Unlike standard Transformers that can be computationally heavy, the Twins-SVT uses a hierarchical structure to extract global features efficiently, allowing the model to "see" the entire tank at once.
  • BiCC (Bi-FPN + CAM Count Module): This is the "aggregator." It uses a Bidirectional Feature Pyramid Network (Bi-FPN) to pass information up and down between high-level and low-level features. It’s paired with a Class Activation Map (CAM) module that forces the network to focus on critical regions where fish are actually present.
  • CRMHead: An adaptive regression head that intelligently fuses features. It recognizes that different layers of the network are better at detecting different things—low-level layers for small, distant fish and high-level layers for large, nearby fish.

Datasets

The researchers utilized two primary data sources:

  1. Lab Tank Dataset: A newly curated dataset consisting of 842 high-resolution images from four different aquaculture tanks. This was a massive manual undertaking: three researchers spent three weeks annotating approximately 205,806 individual fish.
  2. Deep Blue No. 1: To test generalization, the authors used real-world underwater footage from an offshore aquaculture cage in the Yellow Sea. This dataset features Atlantic salmon in much more complex, low-visibility environments compared to the lab tanks.

Key Insights and Ideas

  • Global Context is King: The move to a Transformer backbone (Twins-SVT) was the "aha moment." By capturing global dependencies, the model avoids the "double counting" or "missing" errors common in CNNs when fish are tightly packed.
  • Attention to Detail: The inclusion of the CAM module acts like a spotlight, telling the model which parts of the image are noise (like bubbles or reflections) and which are fish.
  • Scale Agnosticism: By using the CRMHead to adaptively select feature layers, the model doesn't need to be told whether it's looking at a zoomed-in or zoomed-out shot; it automatically weights the features that best match the fish sizes in the frame.

Key Results

FCFormer sets a new benchmark for fish counting in aquaculture:

  • Accuracy: Achieved a peak accuracy of 97.06%.
  • Error Metrics: Recorded a Mean Absolute Error (MAE) of 6.37 and a Root Mean Square Error (MSE) of 8.69.
  • Comparison: It outperformed other state-of-the-art architectures, including CCTrans and DM_Count, showing a 2.02% improvement over the baseline Twins transformer.
  • Real-World Performance: On the "Deep Blue No. 1" ocean dataset, the model maintained a high accuracy of 97.03%, proving it isn't just a "lab-only" solution.

Limitations and Future Work

While FCFormer is highly effective, the authors highlight a few hurdles for the future:

  • Background Diversity: The primary training set had relatively uniform backgrounds. To make the model truly robust, it needs to be exposed to a wider variety of tank materials and water turbidities.
  • Extreme Occlusion: While better than CNNs, the model still faces challenges when fish are so densely packed that they form a solid mass.
  • Edge Deployment: The current model is powerful but computationally demanding. The authors suggest that model compression and acceleration (like pruning or quantization) will be necessary to run FCFormer on resource-constrained edge devices directly at the fish farm.

Summary updated

Abstract

In intelligent feeding recirculating aquaculture system, accurately estimating fish population and density is pivotal for management practices and survival rate assessments. However, challenges arise due to mutual occlusion among fish, rapid movement, and complex breeding environments. Traditional object detection methods based on convolutional neural networks (CNN) often fall short in fully addressing the detection demands for fish schools, especially for distant and small targets. In this regard, we introduce a detection framework dubbed FCFormer (Fish Count Transformer). Specifically, the Twins-SVT backbone network is employed first to extract global features of fish schools. To further enhance feature extraction, especially in the fusion of features at different levels, a Bi-FPN aggregation network model with a CAM Count module is incorporated (BiCC). The CAM module aids in focusing more on critical region features, thus rendering feature fusion more cohesive and effective. Furthermore, to precisely predict density maps and elevate the accuracy of fish counting, we devised an adaptive feature fusion regression head: CRMHead. This approach not only optimizes the feature fusion process but also ensures superior counting precision. Experimental results shown that the proposed FCFormer network achieves an accuracy of 97.06%, with a mean absolute error (MAE) of 6.37 and a root mean square error (MSE) of 8.69. Compared to the Twins transformer, there's a 2.02% improvement, outperforming other transformer-based architectures like CCTrans and DM_Count. The presented FCFormer algorithm can be effectively applied to fish density detection in intelligent feeding recirculating aquaculture system, offering valuable input for the development of intelligent breeding management systems.
BibTeX
@inproceedings{zhu2024fcformer,
  author = {Kaijie Zhu and Xinting Yang and Caiwei Yang and Tingting Fu and Pingchuan Ma and Weichen Hu},
  title = {{FCFormer: fish density estimation and counting in recirculating aquaculture system}},
  booktitle = {Frontiers in Marine Science},
  year = {2024},
  doi = {10.3389/fmars.2024.1370786},
}
Figure