Abstract
This paper considers self-supervised cross-modal coordination as a strategy enabling utilization of multiple modalities and large volumes of unlabeled plankton data to build models for plankton recognition. Automated imaging instruments facilitate the continuous collection of plankton image data on a large scale. Current methods for automatic plankton image recognition rely primarily on supervised approaches, which require labeled training sets that are labor-intensive to collect. On the other hand, some modern plankton imaging instruments complement image information with optical measurement data, such as scatter and fluorescence profiles, which currently are not widely utilized in plankton recognition. In this work, we explore the possibility of using such measurement data to guide the learning process without requiring manual labeling. Inspired by the concepts behind Contrastive Language-Image Pre-training (CLIP), we train encoders for both modalities using only binary supervisory information indicating whether a given image and profile originate from the same particle or from different particles. For plankton recognition, we employ a small labeled gallery of known plankton species combined with a $k$-NN classifier. This approach yields a recognition model that is inherently multimodal, i.e., capable of utilizing information extracted from both image and profile data. We demonstrate that the proposed method achieves high recognition accuracy while requiring only a minimal number of labeled images. Furthermore, we show that the approach outperforms an image-only self-supervised baseline. Code available at https://github.com/Jookare/cross-modal-plankton.
Introduction
Plankton are critical components of the global ecosystem, producing approximately half of the world's oxygen and forming the base of the marine food web. Monitoring their populations is essential for understanding climate change and marine health. However, this is a significant "big data" challenge. Modern instruments like the CytoSense flow cytometer can capture millions of images, but human experts can only label a tiny fraction of them, creating a bottleneck for traditional supervised AI models.
Most current AI models for plankton recognition are image-only and require labor-intensive supervised training. However, instruments like CytoSense also record optical profiles—high-dimensional data on how a particle scatters light and fluoresces across different wavelengths. This research addresses how to leverage this underutilized profile data to build accurate models without needing millions of manually labeled images.
Methodology
The proposed framework is inspired by Contrastive Language-Image Pre-training (CLIP). Instead of matching images to text descriptions, the model matches images to their corresponding optical profiles through a two-stage process.
- Contrastive Pre-training: The model utilizes two separate encoders—one for images (e.g., Vision Transformers or ConvNeXts) and one for profiles (e.g., 1D CNNs or LSTMs). During training, it learns to project both modalities into a shared embedding space. If an image and a profile originate from the same plankton particle, their embeddings are pulled together; if they are from different particles, they are pushed apart. This self-supervised learning requires no human labels.
- $k$-Nearest Neighbor ($k$-NN) Classification: For species recognition, a small "gallery" of labeled samples is used. When a new sample is processed, the model finds its location in the shared embedding space and identifies the species based on the $k$ closest samples in the gallery.
Datasets
The study introduces a new public benchmark, SYKE-Plankton_CytoSense_2025, comprising three multimodal datasets:
- LAB: ~20,000 samples from 24 lab-grown plankton cultures (high-quality data).
- SEA: ~9,000 samples from 38 classes collected in-situ in the Baltic Sea (noisy, real-world data).
- UTO: ~33,000 unlabeled samples from an automated station at Utö Island, used for large-scale self-supervised pre-training.
Key Results and Insights
The cross-modal approach demonstrates significant advantages over traditional methods:
- Multimodal Synergy: The model performs best when utilizing both modalities. Combining image and profile data consistently outperformed using either modality alone, proving they provide complementary information.
- Superior Accuracy: On the LAB dataset, the best configuration (ViT-Tiny + 1D CNN) achieved 96.01% accuracy. It significantly outperformed DINO, a popular image-only self-supervised baseline, reaching ~92% accuracy using only images for inference compared to DINO's ~80%.
- Label Efficiency: High performance was achieved with a gallery of only 16 labeled samples per class, making the system highly practical for field researchers.
- Value of Unlabeled Data: Models pre-trained on raw, unlabeled field data (UTO) generalized better to real-world sea samples than those trained on clean laboratory cultures, highlighting the importance of capturing natural variability.
Conclusion
By treating optical profiles as "natural labels" for images, this cross-modal framework enables the use of vast amounts of unlabeled sensor data for plankton monitoring. While cross-modal alignment remains challenging in complex field environments, the ability to add new species via a small gallery without retraining offers a flexible and efficient path forward for marine biology research. Future work will focus on scaling to even larger datasets and exploring earlier fusion techniques within the neural network architecture.
Summary updated