Cross-Modal Learning for Plankton Recognition

Joona Kareinen, Veikka Immonen, Tuomas Eerola, Lumi Haraguchi, Lasse Lensu, Kaisa Kraft, Sanna Suikkanen, Heikki Kälviäinen

TL;DR

Researchers have adapted the "CLIP" philosophy to marine biology by aligning plankton images with their unique optical "fingerprints" (scatter and fluorescence profiles). This self-supervised approach allows AI models to learn from massive amounts of unlabeled sensor data, achieving high recognition accuracy with only a handful of labeled examples.

What Problem Does This Paper Address?

Plankton are the unsung heroes of our planet, producing roughly half of the world's oxygen. Monitoring their populations is critical for understanding climate change, but it’s a massive "big data" challenge. Modern instruments like the CytoSense flow cytometer can capture millions of images, but human experts can only label a tiny fraction of them.

Most current AI models for plankton recognition are "image-only" and require labor-intensive supervised training. However, instruments like CytoSense also record optical profiles—data on how a particle scatters light and fluoresces across different wavelengths. This paper tackles two main gaps: how to use this underutilized profile data and how to build accurate models without needing millions of manually labeled images.

Methods

The authors propose a two-stage framework inspired by Contrastive Language-Image Pre-training (CLIP). Instead of matching images to text descriptions, they match images to their corresponding optical profiles.

  1. Contrastive Pre-training: The model uses two separate encoders (one for images, one for profiles). During training, it learns to project both into a shared "embedding space." The goal is simple: if an image and a profile come from the same plankton particle, their embeddings should be close together; if they are from different particles, they should be far apart. This is done using self-supervised learning, meaning it requires zero human labels.
  2. $k$-Nearest Neighbor ($k$-NN) Classification: To actually recognize species, the researchers use a small "gallery" of labeled samples. When a new, unknown sample comes in, the model finds its location in the shared embedding space and looks at the $k$ closest samples in the gallery to decide the species.

The authors tested several architectures, including Vision Transformers (ViTs) and ConvNeXts for images, and 1D CNNs and LSTMs for the sequence-based optical profiles.

Datasets

The study utilizes three multimodal datasets collected with CytoSense instruments, which the authors have released as a new public benchmark: SYKE-Plankton_CytoSense_2025.

  • LAB: ~20,000 samples from 24 lab-grown plankton cultures (clean, high-quality data).
  • SEA: ~9,000 samples from 38 classes collected in-situ in the Baltic Sea (noisy, real-world data).
  • UTO: ~33,000 unlabeled samples collected from an automated station at the Utö Island (used for large-scale self-supervised pre-training).

Key Insights and Ideas

  • Profiles as "Natural Labels": The optical profile acts as a high-dimensional "caption" for the image. Because the profile captures physical properties (like chlorophyll content or internal structure) that images might miss, it provides a powerful supervisory signal for free.
  • Multimodal Synergy: The "Aha!" moment is that the model performs best when it uses both modalities. Combining image and profile data consistently outperformed using either one alone, proving that these data types are complementary.
  • The Power of Unlabeled Field Data: Surprisingly, models pre-trained on the raw, unlabeled "UTO" field data generalized better to other real-world sea samples than models trained on "clean" laboratory cultures. This suggests that capturing the natural variability of the ocean is more important than having perfect lab data.
  • Zero-Shot Flexibility: Because the system uses a $k$-NN classifier in a shared space, you can add a new species to the system just by adding a few examples to the gallery—no expensive retraining required.

Key Results

  • Superior Accuracy: On the LAB dataset, the best configuration (ViT-Tiny + 1D CNN) achieved 96.01% accuracy.
  • Beating the Baseline: The cross-modal approach significantly outperformed DINO, a popular image-only self-supervised baseline. For example, on the LAB dataset, the proposed method reached ~92% accuracy using only images for inference, compared to DINO's ~80%.
  • Efficiency: The model achieved high performance with a gallery of only 16 labeled samples per class, making it highly practical for researchers who don't have time to label thousands of images.

Limitations and Future Work

While the results are promising, the authors noted that "cross-modal alignment" (matching an image to a profile it hasn't seen before) is still more difficult in complex field data than in controlled lab data. The performance of the profile-to-image retrieval was lower than image-to-image, suggesting the shared space isn't perfectly aligned yet.

Future work could involve scaling up to even larger unlabeled datasets or exploring more sophisticated "fusion" techniques that combine the modalities earlier in the neural network rather than just at the final voting stage.

Summary updated

Abstract

This paper considers self-supervised cross-modal coordination as a strategy enabling utilization of multiple modalities and large volumes of unlabeled plankton data to build models for plankton recognition. Automated imaging instruments facilitate the continuous collection of plankton image data on a large scale. Current methods for automatic plankton image recognition rely primarily on supervised approaches, which require labeled training sets that are labor-intensive to collect. On the other hand, some modern plankton imaging instruments complement image information with optical measurement data, such as scatter and fluorescence profiles, which currently are not widely utilized in plankton recognition. In this work, we explore the possibility of using such measurement data to guide the learning process without requiring manual labeling. Inspired by the concepts behind Contrastive Language-Image Pre-training, we train encoders for both modalities using only binary supervisory information indicating whether a given image and profile originate from the same particle or from different particles. For plankton recognition, we employ a small labeled gallery of known plankton species combined with a $k$-NN classifier. This approach yields a recognition model that is inherently multimodal, i.e., capable of utilizing information extracted from both image and profile data. We demonstrate that the proposed method achieves high recognition accuracy while requiring only a minimal number of labeled images. Furthermore, we show that the approach outperforms an image-only self-supervised baseline. Code available at https://github.com/Jookare/cross-modal-plankton.
BibTeX
@inproceedings{kareinen2026cross,
  author = {Joona Kareinen and Veikka Immonen and Tuomas Eerola and Lumi Haraguchi and Lasse Lensu and Kaisa Kraft and Sanna Suikkanen and Heikki Kälviäinen},
  title = {{Cross-Modal Learning for Plankton Recognition}},
  booktitle = {ICPR},
  year = {2026},
  doi = {10.1007/978-3-032-31666-0\_25},
}

Loading related content...

Figure