Few-Shot Classification of Fungi Species Using Contrastive Representation Learning and Multimodal Fusion

Lianping Lu, Heng Yang, Shuo Li, Fang Liu, Puhua Chen, Wenping MaCLEF (Working Notes) 2025 · OpenReview

TL;DR

This paper tackles the challenging problem of identifying fungi species with very limited training data (few-shot learning) by combining visual information from images with ecological metadata. They achieve this using a novel two-stage approach involving a custom contrastive loss function and a multimodal fusion architecture, securing second place in the FungiCLEF2025 competition.

What Problem Does This Paper Address?

Accurately identifying fungi species is crucial for biodiversity monitoring and conservation. However, many fungi species are rare, meaning there are very few examples available for training machine learning models. This "few-shot learning" problem is particularly difficult for fine-grained classification, where subtle visual differences distinguish species. Traditional methods struggle when faced with such limited data. The FungiCLEF2025 challenge specifically addresses this by providing a dataset with only 1-4 training instances per species, requiring models to learn from extremely scarce examples. Furthermore, fungi exist within complex ecological contexts, and their identification often relies on more than just visual appearance; factors like location, substrate, and weather can be important clues. This paper aims to bridge this gap by developing a robust few-shot classification system that effectively leverages both visual and contextual (metadata) information.

Methods

The authors propose a two-stage framework:

Stage 1: Visual Feature Learning * DINOv2 Pre-training: They start by using DINOv2, a powerful self-supervised learning model, to extract rich visual features from fungi images. * Transformer Encoder: These features are then fed into a single-layer Transformer encoder with a 16-head attention mechanism. This allows the model to focus on fine-grained visual cues within the images. * Dynamic Weighting Contrastive Loss (DWCL): This is a core innovation. DWCL is an enhanced supervised contrastive loss function designed for few-shot scenarios. It incorporates: * Entropy-based Uncertainty Weighting: The model assigns higher weights to samples that it is less certain about, forcing it to learn from the most challenging examples. * Adaptive Pair Construction: It intelligently selects positive and negative pairs for contrastive learning, ensuring robust intra-class clustering and inter-class separation even with limited data. For categories with very few samples, data augmentation is used to create more training pairs.

Stage 2: Multimodal Fusion and Classification * Metadata Textualization: Ecological metadata (year, month, day, habitat, substrate) is converted into structured text descriptions using a template. * BERT Encoding: These text descriptions are then encoded into numerical representations using BERT. * Q-Former Fusion: The BERT-encoded text features and the refined visual features from Stage 1 are fed into a Q-Former module. This module uses learnable query tokens and multi-head self-attention to fuse the visual and textual information, creating a rich multimodal representation that captures both appearance and ecological context. * Classification Head: Finally, this fused representation is passed through a classification head, trained with a cross-entropy loss, to predict the fungi species.

Datasets

  • FungiCLEF2025 Dataset: This dataset, built from observations submitted to the Atlas of Danish Fungi, is the primary resource. It includes multiple photographs per specimen, along with satellite imagery, meteorological records, and structured metadata. A key characteristic is the extremely limited sample size per class (1-4 observations in the training set).
  • Training set: 4,293 observations, 7,819 images, 2,427 species.
  • Validation set: 1,099 observations, 2,285 images, 570 species.

Key Insights and Ideas

  • Metadata is Crucial for Fine-Grained Fungi: The paper strongly demonstrates that simply relying on visual features is insufficient for accurate fine-grained fungi classification, especially in few-shot settings. Integrating ecological metadata significantly boosts performance by providing complementary contextual information.
  • Contrastive Learning Needs Adaptation for Few-Shot: Standard contrastive learning methods can struggle with highly imbalanced or scarce data. The proposed DWCL, with its uncertainty weighting and adaptive pair construction, is a key enabler for success in this challenging scenario.
  • Transformer Attention for Local Features: The use of a Transformer encoder with multi-head attention effectively captures subtle, locally discriminative visual features that are critical for distinguishing similar fungi species.
  • Q-Former as an Effective Multimodal Bridge: The Q-Former architecture proves adept at fusing information from different modalities (vision and text) by establishing semantic relationships between them.

Key Results

  • Overall Performance: The proposed two-stage framework achieved 2nd place on the FungiCLEF2025 competition leaderboard.
  • Stage 1 Improvement:
    • Only DINOv2 features: Top5 Accuracy of 37.128%.
  • DINOv2 + Transformer: Accuracy improved to 70.892%.
  • DINOv2 + Transformer + Metadata: Accuracy further boosted to 76.991%.
  • DWCL vs. Standard Contrastive Loss: The proposed DWCL achieved 76.991% Top5 Accuracy, outperforming the standard contrastive loss function (74.778%).
  • Ablation Studies:
    • A single-layer Transformer architecture was optimal, avoiding overfitting.
  • 16 attention heads performed better than 32 heads.
  • Peak validation performance was achieved at 100 training epochs, with degradation beyond that.
  • Final Submission: An ensemble of top-performing models (using weighted voting) achieved an aggregated score of 78.137%.

Limitations and Future Work

  • Overfitting with Deep Architectures: The study found that deeper Transformer architectures led to overfitting on the small-scale fungi dataset, highlighting the need for careful model selection in few-shot scenarios.
  • Early Stopping Necessity: Performance degraded after 100 epochs, emphasizing the importance of early stopping strategies.
  • Future Directions: The authors suggest future research will focus on:
  • Satellite Data Augmentation: Incorporating satellite imagery as an additional data source.
  • Explainable Attention Mechanisms: Developing methods to understand why the model makes certain predictions, which is crucial for practical field applications.

Summary updated

BibTeX
@inproceedings{lu2025few,
  author = {Lianping Lu and Heng Yang and Shuo Li and Fang Liu and Puhua Chen and Wenping Ma},
  title = {{Few-Shot Classification of Fungi Species Using Contrastive Representation Learning and Multimodal Fusion}},
  booktitle = {CLEF (Working Notes)},
  year = {2025},
  url = {https://openreview.net/forum?id=squ8G3iqdV},
}
Figure