Ensembles of Vision Transformers as a New Paradigm for Automated Classification in Ecology

Sreenath P. Kyathanahally, Thomas Hardeman, Marta Reyes, Ewa Merz, Thea Bulas, Francesco Pomati, Marco Baity-Jesi

Introduction

Automated biodiversity monitoring is a critical component of modern conservation efforts, yet it remains a significant technical challenge. While deep learning—specifically Convolutional Neural Networks (CNNs)—has advanced the field, these models often struggle with the inherent noise, class imbalance, and "measurement noise" found in ecological field data. This paper introduces EDeiTs (Ensembles of Data-efficient image Transformers), a novel approach that leverages the unique properties of Vision Transformers to establish a new state-of-the-art (SOTA) in ecological classification.

The EDeiT Framework

The proposed method utilizes the DeiT-Base architecture, a variant of the Vision Transformer (ViT) optimized for smaller datasets through a distillation procedure. The core innovation lies in the ensembling strategy: rather than selecting a single optimal model, the researchers train multiple versions of the same architecture with varying initial weights.

To combine model predictions, the study compared two primary techniques:

  1. Arithmetic Mean (Sum Rule): A standard averaging of confidence vectors.
  2. Geometric Mean (Product Rule): A more stringent "veto" mechanism that penalizes classes where any single model shows low confidence. This approach proved particularly effective for DeiTs, which exhibit a less peaked confidence profile than CNNs.

Experimental Results

The EDeiT approach was rigorously validated across ten diverse ecological datasets, including coral reefs (RSMAS, EILAT), plankton (ZooLake, WHOI, Kaggle, ZooScan), birds (NA-Birds), and terrestrial animals (Stanford Dogs, Sri Lankan Beetles, Florida Wildtrap).

Comparison of error rates between current state-of-the-art (SOTA) models and EDeiT ensembles across ten ecological datasets.

As illustrated in the results, EDeiTs outperformed the previous SOTA on every dataset tested. Error reductions ranged from approximately 29% for NA-Birds to a total elimination of error (100% reduction) in the RSMAS coral dataset. On the complex ZooLake plankton dataset, the ensemble achieved an F1-score of 0.983, misclassifying only 15 out of 2,691 images—most of which belonged to ambiguous "junk" categories.

Why Transformers Excel in Ensembles

A key finding of this research is that while individual DeiT models may not always significantly outperform individual CNNs, they are far superior "team players." CNNs tend to be overconfident and often share the same failure modes. In contrast, DeiT models exhibit a higher degree of disagreement in their errors. This independence allows the ensemble to correct individual mistakes, making the collective decision significantly more robust.

Interpretability and Transparency

For ecologists, understanding the rationale behind a model's decision is as important as the accuracy of the decision itself. Unlike the "black box" nature of many CNNs, the attention mechanism in Transformers provides inherent interpretability.

Attention maps demonstrating the Transformer's ability to localize key morphological features of various organisms.

By generating attention maps, researchers can visualize the specific regions—such as the wing of a bird or the legs of a beetle—that the model prioritized. This transparency builds trust and allows for the verification of biological relevance in automated systems, ensuring that models are learning meaningful features rather than background noise.

Conclusion and Future Directions

The transition from CNNs to ensembles of Vision Transformers represents a significant shift in ecological computer vision. By reducing error rates and providing interpretable results, EDeiTs offer a more robust path toward fully autonomous biodiversity monitoring. Future research will focus on addressing the computational demands of these large models and testing their resilience against "data shift" in uncontrolled, real-world field environments.

Summary updated

BibTeX
@inproceedings{kyathanahally2022ensembles,
  author = {Sreenath P. Kyathanahally and Thomas Hardeman and Marta Reyes and Ewa Merz and Thea Bulas and Francesco Pomati and Marco Baity-Jesi},
  title = {{Ensembles of Vision Transformers as a New Paradigm for Automated Classification in Ecology}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-21910-0},
}
Figure