Abstract
This research introduces the Cross-Covariance Attended Compact Feed-Forward Sequential Memory Network (CC-FSMN), a specialized neural network architecture designed for the acoustic monitoring of 32 marine mammal species. By combining sequential memory blocks with efficient cross-covariance attention, the model achieves a state-of-the-art 0.931 F1 score while training significantly faster than traditional Convolutional Neural Networks (CNNs). The CC-FSMN offers a lightweight solution for large-scale bioacoustic classification, outperforming standard models like VGG16 in both accuracy and computational efficiency.
Introduction
Human activities, particularly military sonar and commercial shipping, pose significant threats to marine mammal populations. To mitigate these impacts, researchers use hydrophones to monitor animal vocalizations. However, manual identification is impossible at scale, and existing deep learning solutions face two major hurdles: high computational requirements and limited species scope. Most current models use CNNs (e.g., ResNet or VGG), which treat audio as images and require substantial hardware resources. This paper proposes the CC-FSMN, a lightweight architecture designed to handle a diverse range of 32 species without needing massive computing clusters.
Methodology
The classification pipeline begins with signal resampling and segmentation, followed by Cepstral Mean and Variance Normalization (CMVN). Features are extracted using Mel-scale frequency cepstral coefficients (MFCCs), which represent the acoustic texture of the sound.
The core of the system is the CC-FSMN architecture, which integrates two key components:
- Compact Feed-Forward Sequential Memory Network (cFSMN): Unlike Recurrent Neural Networks (RNNs), FSMNs use memory blocks to capture long-term dependencies within a feed-forward structure, making them highly efficient to train.
- Cross-Covariance Attention (XCA): Standard self-attention mechanisms scale quadratically with input length. XCA generates attention maps from a cross-covariance matrix instead, providing robustness against noise and significantly reducing the computational cost.
Results
The model was evaluated using the Watkins Marine Mammal Sound Database (WMMSD), encompassing 1,694 sound cuts across 32 species. The CC-FSMN was compared against industry standards including AlexNet, ResNet50, VGG16, and DenseNet.
As shown in Figure 3, the CC-FSMN demonstrates exceptional computational efficiency. It utilizes only ~8.4 million parameters and completes training in just 11 minutes—a 23x speedup compared to VGG16's 371 minutes. It also achieved the fastest inference time at 3.3 seconds.
In terms of accuracy, CC-FSMN outperformed all other models with an F1 score of 0.931 and a recall of 0.957. This suggests that architectures designed for sequential data, like FSMNs, are fundamentally better suited for bioacoustics than image-centric CNNs.
Discussion and Future Work
A notable insight from the study is that adding the XCA layer actually reduced training time, likely by helping the model reach its optimal state faster. Despite its high performance, the model struggled to distinguish between Sperm whales and Short-finned pilot whales, likely due to similarities in their MFCC features.
Future work will focus on testing the model against noisier real-world data and investigating X-vector embeddings to improve differentiation between similar-sounding species. This research paves the way for more accessible and efficient marine life monitoring systems.
Summary updated