Ensembles of data-efficient vision transformers as a new paradigm for automated classification in ecology

Sreenath P. Kyathanahally, Thomas Hardeman, Marta Reyes, Ewa Merz, Thea Bulas, Philipp Brun, Francesco Pomati, Marco Baity‐Jesi

Abstract

Monitoring biodiversity is paramount to manage and protect natural resources. Collecting images of organisms over large temporal or spatial scales is a promising practice to monitor the biodiversity of natural ecosystems, providing large amounts of data with minimal interference with the environment. Deep learning models are currently used to automate classification of organisms into taxonomic units. However, imprecision in these classifiers introduces a measurement noise that is difficult to control and can significantly hinder the analysis and interpretation of data. We overcome this limitation through ensembles of Data-efficient image Transformers (DeiTs), which not only are easy to train and implement, but also significantly outperform the previous state of the art (SOTA). We validate our results on ten ecological imaging datasets of diverse origin, ranging from plankton to birds. On all the datasets, we achieve a new SOTA, with a reduction of the error with respect to the previous SOTA ranging from 29.35% to 100.00%, and often achieving performances very close to perfect classification. Ensembles of DeiTs perform better not because of superior single-model performances but rather due to smaller overlaps in the predictions by independent models and lower top-1 probabilities. This increases the benefit of ensembling, especially when using geometric averages to combine individual learners.

Introduction

Ecologists are increasingly deploying automated imaging systems—from underwater cameras to forest camera traps—to monitor biodiversity. While these systems generate massive amounts of data, the primary bottleneck remains accurate classification. Current deep learning standards, primarily Convolutional Neural Networks (CNNs), often require extensive hyperparameter tuning to reach high accuracy. Even then, "measurement noise" from misclassifications can lead to incorrect ecological conclusions about species abundance and interactions. This paper introduces Ensembles of Data-efficient image Transformers (EDeiTs) as a robust, "plug-and-play" paradigm to handle diverse ecological data with minimal manual intervention.

Methodology

The researchers utilized Data-efficient image Transformers (DeiTs), which employ a distillation procedure where a transformer (the student) learns from a pre-trained CNN (the teacher). This allows them to be trained effectively on smaller datasets compared to standard Vision Transformers (ViTs).

The core innovation lies in the use of ensembles (EDeiTs). Multiple DeiT-Base models were trained with different initial weight configurations. Their predictions were combined using two primary techniques:

  • Arithmetic Mean: Standard averaging of model outputs.
  • Geometric Mean: Acts as a "veto mechanism," significantly penalizing classes where even one model has very low confidence.

All models were fine-tuned from ImageNet-pre-trained weights using a simple, fixed training schedule, intentionally avoiding dataset-specific hyperparameter tuning to demonstrate generalizability.

Key Results

The EDeiT approach was evaluated on ten publicly available ecological datasets, including coral reef patches (RSMAS, EILAT), various plankton datasets (ZooLake, WHOI, Kaggle, ZooScan), North American birds (NA-Birds), dog breeds (Stanford Dogs), tiger beetles, and Florida wildlife camera traps.

Comparison of EDeiT error rates against current state-of-the-art (SOTA) across ten ecological datasets.

As shown in the results, EDeiTs achieved a new SOTA on all ten datasets. The reduction in error rates relative to previous benchmarks ranged from 29.35% to 100%, often achieving near-perfect classification. Notably, these results were achieved without dataset-specific optimization, suggesting EDeiTs can be deployed as a standard tool for ecologists without requiring deep expertise in deep learning.

Key Insights: The Power of Disagreement

The success of EDeiTs is not due to individual models being inherently "smarter" than CNNs, but rather due to the nature of their errors. Independent DeiT models tend to disagree with each other more often when they are wrong. While CNNs often "fail together" on the same features, DeiTs provide more diverse "opinions," making the ensemble collective much more robust.

Analysis of prediction overlaps (Right/Wrong) between ensemble members for CNNs vs. DeiTs.

Furthermore, DeiT models tend to have less "peaked" top-1 probabilities compared to CNNs. This lower over-confidence makes them better candidates for ensembling, as it allows the collective vote to more effectively correct individual errors. The use of geometric averaging was particularly effective; if one model is certain an image is not a specific species (near-zero confidence), the geometric mean effectively vetoes that class, leading to higher precision.

Conclusion and Future Work

EDeiTs represent a powerful solution for specialized ecological tasks, offering superior performance and better interpretability through attention mechanisms. While they may not necessarily outperform architectures heavily over-engineered for mainstream benchmarks like ImageNet, their robustness makes them ideal for real-world ecological field deployments. Future work will explore how EDeiTs handle "data shift," such as seasonal or locational variations, where preliminary evidence suggests Transformers may be more resilient than traditional CNNs.

Summary updated

Abstract

Monitoring biodiversity is paramount to manage and protect natural resources. Collecting images of organisms over large temporal or spatial scales is a promising practice to monitor the biodiversity of natural ecosystems, providing large amounts of data with minimal interference with the environment. Deep learning models are currently used to automate classification of organisms into taxonomic units. However, imprecision in these classifiers introduces a measurement noise that is difficult to control and can significantly hinder the analysis and interpretation of data. We overcome this limitation through ensembles of Data-efficient image Transformers (DeiTs), which not only are easy to train and implement, but also significantly outperform the previous state of the art (SOTA). We validate our results on ten ecological imaging datasets of diverse origin, ranging from plankton to birds. On all the datasets, we achieve a new SOTA, with a reduction of the error with respect to the previous SOTA ranging from 29.35% to 100.00%, and often achieving performances very close to perfect classification. Ensembles of DeiTs perform better not because of superior single-model performances but rather due to smaller overlaps in the predictions by independent models and lower top-1 probabilities. This increases the benefit of ensembling, especially when using geometric averages to combine individual learners. While we only test our approach on biodiversity image datasets, our approach is generic and can be applied to any kind of images.
BibTeX
@inproceedings{kyathanahally2022ensembles,
  author = {Sreenath P. Kyathanahally and Thomas Hardeman and Marta Reyes and Ewa Merz and Thea Bulas and Philipp Brun and Francesco Pomati and Marco Baity‐Jesi},
  title = {{Ensembles of data-efficient vision transformers as a new paradigm for automated classification in ecology}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-21910-0},
}
Figure