Introduction
In the controlled environment of a laboratory, machine learning models for species identification typically operate under a "closed-world" assumption. This assumption presupposes that every specimen encountered during deployment belongs to one of the specific categories present in the training dataset. However, in real-world biodiversity monitoring, this assumption is fundamentally flawed. With an estimated 90% of Earth’s species currently unknown to science and many described species lacking sufficient photographic data for training, models frequently encounter novel organisms.
When a standard closed-set model encounters a species it has never seen, it does not signal uncertainty; instead, it confidently misclassifies the specimen into a known category. This behavior presents a significant barrier to automated biodiversity monitoring and the discovery of new species. To address this, researchers have turned to Open-Set Recognition (OSR)—the ability of a model to accurately classify known categories while simultaneously identifying and flagging "unknowns" as outliers. This paper introduces Open-Insect, a massive new benchmark designed to evaluate OSR algorithms in the context of fine-grained, geographically diverse biological data.
The Open-Insect Benchmark
Existing OSR benchmarks often rely on generic datasets like ImageNet, which fail to capture the nuances of the natural world, such as extreme visual similarity between species and geographic complexities. Open-Insect was curated from the AMI-GBIF dataset and other public records to provide a more rigorous testing ground.
Dataset Composition
The benchmark focuses on moths due to their high diversity, visual similarity, and frequent capture in automated camera traps. The data is organized into three distinct geographic splits:
- Northeastern North America
- Western Europe
- Central America
To test the limits of model discrimination, the researchers defined three types of "unknown" inputs:
- Local moths: Species from the same geographic region that were excluded from the training set. These are the most difficult to detect due to taxonomic and visual proximity to known species.
- Non-local moths: Species from entirely different continents (e.g., Australia).
- Non-moths: Other arthropods, such as spiders or beetles, which represent the easiest category for outlier detection.
Methodology
The study conducted a comprehensive sweep of 38 OSR algorithms, categorized into three primary approaches:
- Post-hoc Methods: These methods apply a scoring function, such as Maximum Softmax Probability (MSP), to the outputs of a pre-trained classifier to identify outliers without requiring additional training.
- Training-time Regularization: These algorithms modify the training process—often through specialized loss functions—to create more compact clusters for known classes, making outliers easier to distinguish.
- Training with Auxiliary Data: These models are exposed to "known unknowns" (images of species outside the target set) during training to learn the characteristics of outliers. The authors introduced two simple baselines in this category: NovelBranch and Extended, which treat auxiliary data as either a single outlier class or multiple distinct new classes to sharpen decision boundaries.
Key Insights and Results
The benchmarking process revealed several critical findings for the future of AI-driven biodiversity monitoring:
The Challenge of Semantic Shift
Models performed significantly worse at detecting unknown species from the same geographic region compared to those from distant regions. This is attributed to the fact that "local" unknowns are often taxonomically related (sharing the same genus or family) to the training data, resulting in a subtle visual "semantic shift" that eludes many algorithms.
The Efficacy of Simple Baselines
Surprisingly, the Maximum Softmax Probability (MSP)—one of the simplest post-hoc methods—remained highly competitive. Many complex modern algorithms failed to significantly outperform this baseline in fine-grained scenarios, suggesting that architectural complexity does not always translate to better OSR performance in biology.
The Role of Foundation Models and Data
The use of BioCLIP, a foundation model specifically trained on biological data, consistently outperformed standard ImageNet pre-training. Furthermore, while auxiliary data improved performance, the quality of that data mattered; training on diverse, related insect species provided more benefit than training on generic datasets like TinyImageNet.
Performance Metrics
The top-performing methods achieved over 85% AUROC (Area Under the Receiver Operating Characteristic curve) across all regions. In high-biodiversity, data-poor regions like Central America, the use of auxiliary data was particularly impactful, boosting performance by over 3.2 percentage points.
Discussion and Future Work
While the results suggest that OSR is reaching a level of maturity suitable for real-world pilot applications, several limitations remain. The current benchmark utilizes high-resolution images, whereas real-world camera trap data is often plagued by motion blur, poor lighting, and environmental clutter. Additionally, the computational cost of training-time regularization methods limited the ability to perform multiple training runs at scale.
Future research must expand beyond moths to other hyper-diverse and understudied taxa, such as flies, beetles, and Hymenoptera. By providing a standardized proving ground, Open-Insect facilitates the development of robust AI tools capable of identifying the "needles in the haystack"—the rare or undiscovered species essential for understanding global biodiversity loss.
Summary updated