When Large Kernel Meets Vision Transformer: A Solution for SnakeCLEF & FungiCLEF

Yang Shen, Xuhao Sun, Zijian ZhuCLEF (Working Notes) 2022 · OpenReview

TL;DR

This paper tackles the challenges of fine-grained species identification in the SnakeCLEF and FungiCLEF competitions, specifically addressing issues like long-tailed class distributions and open-set recognition. The authors propose a novel "CoLKANet" architecture that combines large kernel convolutions with Vision Transformers, achieving strong results on both challenging benchmarks.

What Problem Does This Paper Address?

Accurately identifying species from images is crucial for biodiversity conservation and ecological research. However, this task is made significantly harder by several factors:

  • Fine-grained recognition: Distinguishing between visually similar species requires models to capture subtle differences.
  • Long-tailed distributions: In real-world datasets, a few species are very common, while most are rare. This imbalance can bias models towards common classes, leading to poor performance on rare ones.
  • Open-set recognition: The test data may contain species not present in the training set, requiring models to identify unknown instances rather than misclassifying them.

The LifeCLEF lab aims to advance solutions for these problems through competitions like SnakeCLEF and FungiCLEF. This paper presents the authors' approach to these challenges, focusing on robust identification systems for snakes and fungi using images and metadata.

Methods

The core of the paper's contribution lies in its proposed architecture and the techniques employed to handle the competition's challenges:

  • CoLKANet: This novel backbone architecture is a hybrid model that combines the strengths of large kernel convolutions and Vision Transformers (ViTs). The idea is that large kernels can help ViTs "see better" by capturing broader contextual information, while ViTs excel at global feature representation. The paper references existing work on large kernel design and attention mechanisms in computer vision.
  • Handling Long-Tailed Distributions: The authors employ "label-aware smoothing" as a loss function. This technique adjusts the learning process to give more importance to tail classes (rare species) and mitigate the over-confidence of models on head classes (common species).
  • Ensembling: To maximize performance, the authors utilized an ensembling strategy, combining predictions from multiple trained models. This approach aims to leverage the diversity of different architectures and training configurations.
  • Other Techniques: The paper also details various standard and advanced techniques used, including:
  • Augmentation: Extensive data augmentation strategies (RandomResizedCrop, TrivialAugment, CutMix, Mixup, etc.) were used for regularization.
  • Test Time Augmentation (TTA): Cropping test images multiple times and averaging predictions to improve robustness.
  • Pseudo-Labeling: Generating labels for unlabeled test data to augment the training set (though this was found to be less effective for FungiCLEF).
  • Weight Decay Tuning and EMA: Standard regularization and model averaging techniques.
  • FixRes: A technique to address the discrepancy between training and testing image resolutions, primarily for CNNs.

Datasets

The paper focuses on two datasets from the LifeCLEF 2022 evaluation campaign:

  • SnakeCLEF 2022: This dataset comprises 187,129 snake observations with 318,532 photographs across 1,572 species from 208 countries, sourced from iNaturalist. It exhibits a significant long-tailed distribution.
  • FungiCLEF 2022: This dataset is based on the Danish Fungi 2020 dataset, containing 295,938 training images of 1,604 species. It includes rich metadata (habitat, substrate, time, location) and also presents a long-tailed distribution. The test set contains 59,420 observations with 118,676 images and 3,134 species.

Key Insights and Ideas

  • Synergy of Large Kernels and Transformers: The paper strongly suggests that combining large kernel convolutions with Vision Transformers is a powerful paradigm for fine-grained image recognition, potentially addressing limitations of each individual approach.
  • Label-Aware Smoothing is Crucial for Long-Tailed Data: The authors found that directly applying standard cross-entropy loss is insufficient for long-tailed datasets. Label-aware smoothing proved effective in improving performance by better handling the class imbalance.
  • Public vs. Private Leaderboard Discrepancy: The paper highlights a significant gap between public and private leaderboard scores, suggesting that strategies optimized for the public leaderboard (e.g., aggressive pseudo-labeling or specific thresholding) might not generalize well. This points to potential differences in data distribution or evaluation criteria.
  • Open-Set Strategy for FungiCLEF: For the FungiCLEF competition, the authors discovered that a significant portion of the test data likely belonged to an "open set" (species not in the training set). They found that a simple thresholding strategy, rather than complex meta-data analysis, was more effective for this specific challenge.

Key Results

The authors achieved competitive results in both competitions:

  • SnakeCLEF: A final score of 85.4% Macro F1-Score on the private leaderboard.
  • FungiCLEF: A final score of 78.9% Macro F1-Score on the private leaderboard.

The paper also presents detailed results for various backbone models and ensembles, showing the incremental improvements gained from different techniques. For instance, their proposed CoLKANet backbone achieved a score of 80.1% on SnakeCLEF, and an ensemble including it reached 85.4%.

Limitations and Future Work

  • Pseudo-Labeling Effectiveness: While pseudo-labeling improved scores on the SnakeCLEF public leaderboard, it was not effective for FungiCLEF, possibly due to the open-set nature of that competition.
  • Public vs. Private Leaderboard Overfitting: The authors acknowledge that their strategy of fitting the public leaderboard might have led to overfitting, as evidenced by the score drop on the private leaderboard for some approaches.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the CoLKANet architecture suggests further exploration of hybrid models combining convolutional and transformer components. Additionally, more robust methods for handling open-set recognition in complex, real-world scenarios remain an active area of research.

Summary updated

BibTeX
@inproceedings{shen2022when,
  author = {Yang Shen and Xuhao Sun and Zijian Zhu},
  title = {{When Large Kernel Meets Vision Transformer: A Solution for SnakeCLEF \& FungiCLEF}},
  booktitle = {CLEF (Working Notes)},
  year = {2022},
  url = {https://openreview.net/forum?id=t18y9EU77F},
}
Figure