Open-Set Recognition of Novel Species in Biodiversity Monitoring

Yuyan Chen, Nico Lang, B. Christian Schmidt, Aditya Jain, Yves Basset, Sara Beery, Maxim Larrivée, David Rolnick

Introduction

In the controlled environment of a laboratory, machine learning models for species identification typically operate under a "closed-world" assumption. This assumption presupposes that every specimen encountered during deployment belongs to one of the specific categories present in the training dataset. However, in real-world biodiversity monitoring, this assumption is fundamentally flawed. With an estimated 90% of Earth’s species currently unknown to science and many described species lacking sufficient photographic data for training, models frequently encounter novel organisms.

When a standard closed-set model encounters a species it has never seen, it does not signal uncertainty; instead, it confidently misclassifies the specimen into a known category. This behavior presents a significant barrier to automated biodiversity monitoring and the discovery of new species. To address this, researchers have turned to Open-Set Recognition (OSR)—the ability of a model to accurately classify known categories while simultaneously identifying and flagging "unknowns" as outliers. This paper introduces Open-Insect, a massive new benchmark designed to evaluate OSR algorithms in the context of fine-grained, geographically diverse biological data.

The Open-Insect Benchmark

Existing OSR benchmarks often rely on generic datasets like ImageNet, which fail to capture the nuances of the natural world, such as extreme visual similarity between species and geographic complexities. Open-Insect was curated from the AMI-GBIF dataset and other public records to provide a more rigorous testing ground.

Dataset Composition

The benchmark focuses on moths due to their high diversity, visual similarity, and frequent capture in automated camera traps. The data is organized into three distinct geographic splits:

  • Northeastern North America
  • Western Europe
  • Central America

To test the limits of model discrimination, the researchers defined three types of "unknown" inputs:

  1. Local moths: Species from the same geographic region that were excluded from the training set. These are the most difficult to detect due to taxonomic and visual proximity to known species.
  2. Non-local moths: Species from entirely different continents (e.g., Australia).
  3. Non-moths: Other arthropods, such as spiders or beetles, which represent the easiest category for outlier detection.

Methodology

The study conducted a comprehensive sweep of 38 OSR algorithms, categorized into three primary approaches:

  1. Post-hoc Methods: These methods apply a scoring function, such as Maximum Softmax Probability (MSP), to the outputs of a pre-trained classifier to identify outliers without requiring additional training.
  2. Training-time Regularization: These algorithms modify the training process—often through specialized loss functions—to create more compact clusters for known classes, making outliers easier to distinguish.
  3. Training with Auxiliary Data: These models are exposed to "known unknowns" (images of species outside the target set) during training to learn the characteristics of outliers. The authors introduced two simple baselines in this category: NovelBranch and Extended, which treat auxiliary data as either a single outlier class or multiple distinct new classes to sharpen decision boundaries.

Key Insights and Results

The benchmarking process revealed several critical findings for the future of AI-driven biodiversity monitoring:

The Challenge of Semantic Shift

Models performed significantly worse at detecting unknown species from the same geographic region compared to those from distant regions. This is attributed to the fact that "local" unknowns are often taxonomically related (sharing the same genus or family) to the training data, resulting in a subtle visual "semantic shift" that eludes many algorithms.

The Efficacy of Simple Baselines

Surprisingly, the Maximum Softmax Probability (MSP)—one of the simplest post-hoc methods—remained highly competitive. Many complex modern algorithms failed to significantly outperform this baseline in fine-grained scenarios, suggesting that architectural complexity does not always translate to better OSR performance in biology.

The Role of Foundation Models and Data

The use of BioCLIP, a foundation model specifically trained on biological data, consistently outperformed standard ImageNet pre-training. Furthermore, while auxiliary data improved performance, the quality of that data mattered; training on diverse, related insect species provided more benefit than training on generic datasets like TinyImageNet.

Performance Metrics

The top-performing methods achieved over 85% AUROC (Area Under the Receiver Operating Characteristic curve) across all regions. In high-biodiversity, data-poor regions like Central America, the use of auxiliary data was particularly impactful, boosting performance by over 3.2 percentage points.

Discussion and Future Work

While the results suggest that OSR is reaching a level of maturity suitable for real-world pilot applications, several limitations remain. The current benchmark utilizes high-resolution images, whereas real-world camera trap data is often plagued by motion blur, poor lighting, and environmental clutter. Additionally, the computational cost of training-time regularization methods limited the ability to perform multiple training runs at scale.

Future research must expand beyond moths to other hyper-diverse and understudied taxa, such as flies, beetles, and Hymenoptera. By providing a standardized proving ground, Open-Insect facilitates the development of robust AI tools capable of identifying the "needles in the haystack"—the rare or undiscovered species essential for understanding global biodiversity loss.

Summary updated

Abstract

Machine learning is increasingly being applied to facilitate long-term, large-scale biodiversity monitoring. With most species on Earth still undiscovered or poorly documented, species-recognition models are expected to encounter new species during deployment. We introduce Open-Insects, a fine-grained image recognition benchmark dataset for open-set recognition and out-of-distribution detection in biodiversity monitoring. Open-Insects makes it possible to evaluate algorithms for new species detection on several geographical open-set splits with varying difficulty. Furthermore, we present a test set recently collected in the wild with 59 species that are likely new to science. We evaluate a variety of open-set recognition algorithms, including post-hoc methods, training-time regularization, and training with auxiliary data, finding that the simple post-hoc approach of utilizing softmax scores remains a strong baseline. We also demonstrate how to leverage auxiliary data to improve the detection performance when the training dataset is limited. Our results provide timely insights to guide the development of computer vision methods for biodiversity monitoring and species discovery.
BibTeX
@inproceedings{chen2025open,
  author = {Yuyan Chen and Nico Lang and B. Christian Schmidt and Aditya Jain and Yves Basset and Sara Beery and Maxim Larrivée and David Rolnick},
  title = {{Open-Set Recognition of Novel Species in Biodiversity Monitoring}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=63Tia99ofI},
}
Figure