Abstract
Large-scale biodiversity datasets, such as those from iNaturalist, are increasingly used for fine-grained visual classification. However, these datasets are "opportunistic," collected by volunteers with inherent biases. The DivShift framework provides a systematic approach to quantify how these real-world biases—spatial, temporal, taxonomic, observer, and sociopolitical—impact machine learning model performance. Using a new dataset of 7.5 million images, we demonstrate that while models show some resilience, specific shifts like observer engagement and human geographic footprints significantly degrade accuracy.
Introduction: The Challenge of Opportunistic Data
Citizen science has revolutionized biodiversity monitoring, providing a wealth of data for AI development. Yet, this data is non-randomly sampled. Volunteers tend to photograph species near urban centers (spatial bias), during weekends or specific events (temporal bias), and focus on "charismatic" species (taxonomic bias). The critical question is how these sampling strategies affect the reliability of AI models in real-world ecological applications. If a model is trained primarily on suburban observations, its performance on rare species in remote wilderness remains uncertain.
Methodology: The DivShift Framework
The DivShift framework treats domain-specific biases as distribution shifts. The process follows three main steps:
- Partitioning: The dataset is split into subsets ($P_A$ and $P_B$) based on expert-verified biases, such as "Wilderness" vs. "Human-Modified" land.
- Quantifying Shift: The Jensen-Shannon Distance (JSD) is employed to measure the divergence in label distributions between these partitions.
- Performance Evaluation: Models (e.g., ResNet, ViT) are trained on one partition and tested on others to measure the "performance gap."
Partitions are categorized as strongly biased if the drop in accuracy exceeds the label shift, or weakly biased if the model generalizes better than the shift would suggest.
The DivShift-NAWC Dataset
To support this research, we introduce DivShift-NAWC (North American West Coast). This dataset contains nearly 7.5 million images covering over 7,500 plant species across seven terrestrial biomes. The data is partitioned across five primary bias types:
- Spatial: Wilderness vs. Human-Modified areas based on the Global Human Footprint Index.
- Temporal: Observations during the "City Nature Challenge" vs. standard periods.
- Taxonomic: Naturally long-tailed vs. artificially balanced species distributions.
- Observer: "Engaged" power users (>1,000 observations) vs. "Casual" users (<50 observations).
- Sociopolitical: Shifts across state and national borders (e.g., California vs. Sonora).
Key Insights and Results
Our analysis reveals several critical findings regarding model generalization:
The "Human Footprint" Asymmetry
Models trained on human-modified areas generalize well to wilderness. Conversely, wilderness-trained models struggle in modified environments (an 11.3% drop in Top-1 accuracy). This suggests that wilderness data may be too sparse to capture the full phenotypic diversity present in more populated regions.
Observer Engagement and Data Quality
The most dramatic performance gap involves observer engagement. Models trained on data from "engaged" observers provide a +4.9% boost when tested on casual data. However, models trained on "casual" data see a 23.1% accuracy drop when tested on engaged data, indicating that power users provide significantly higher-quality training signals.
The Impact of Sociopolitical Boundaries
Interestingly, sociopolitical boundaries (state lines) often create sharper distribution shifts than ecological boundaries. This highlights how human policy and regional data availability significantly shape the AI's perception of the natural world.
Model Scaling
While larger architectures like Vision Transformers (ViT) improve absolute accuracy by 15-20% over ResNet-18, the relative performance drops caused by distribution shifts remain consistent. This suggests that architectural improvements alone are insufficient to overcome fundamental data biases.
Limitations and Future Directions
While DivShift identifies performance drops, it does not yet explain the underlying causal mechanisms (e.g., lighting vs. morphology). Future work will explore self-supervised learning, implement per-observation data splitting to prevent leakage, and move toward measuring shifts in "environmental space" (e.g., climate hulls) to better capture ecological nuances.
Summary updated