A Computer Vision Method for Finding Mislabelled Specimens Within Natural History Collections

Jack D. Hollister, G. Martin, Xiaohao Cai, Tammy Horton, Olivia Powell, Mark Sterling, Glory Turnbull, Benjamin Price, Phillip B. FenbergEcology 2025 · DOI

TL;DR

This paper introduces a computer vision (CV) pipeline to automatically identify mislabelled specimens in large natural history collections. By flagging specimens that the CV model consistently misidentifies, the system significantly aids human experts in prioritizing verification efforts, leading to more accurate and efficiently managed collections.

What Problem Does This Paper Address?

Natural history collections (NHCs) are invaluable resources for understanding biodiversity and evolution, but they face significant management challenges. The sheer volume of specimens, coupled with shrinking budgets and declining taxonomic expertise, leads to a high rate of mislabelled or missing specimens. While genetic analysis offers precise identification, it's often too costly and destructive for large-scale application. Traditional visual identification is time-consuming and relies on scarce expertise. This paper tackles the problem of efficiently and accurately verifying the vast number of specimens in digitized NHCs, aiming to improve data integrity and preserve these collections for future research.

Methods

The core of this research is a computer vision pipeline designed to detect specimens whose visual appearance doesn't match their assigned label. The pipeline utilizes a VGG16 convolutional neural network, pre-trained on ImageNet, for initial feature learning. This base model is then fine-tuned on a custom dataset of digitized butterfly and moth specimens.

A key methodological innovation is the use of a "Reoccurring Prediction Value" (RPV). The pipeline is run 100 times on the test dataset, and for each specimen, an RPV is calculated based on how consistently the model predicts a species different from its label. Specimens with a high RPV (e.g., consistently flagged across many runs) are considered strong candidates for mislabelling.

To further refine the process and address potential issues with the CV model focusing on labels rather than specimens, a YOLOv8 object detection algorithm was employed to crop images, ensuring the model focuses on the specimen itself. The pipeline also incorporates a multi-stage verification process: initial CV flagging, followed by expert visual inspection, and finally, genetic verification for a subset of specimens.

Datasets

The study utilizes the digitized British and Irish Lepidoptera collection from the Natural History Museum (NHM), London, specifically the "iCollections" dataset.

  • Training and Validation Data: The dataset was split into training, validation, and testing sets. For each species class, 300 images were randomly sampled, with 250 for training and 50 for validation. These images were synthetically augmented (rotations, zooms, brightening) to improve model robustness.
  • Filtered Datasets: Species with fewer than 400 specimens were excluded to ensure sufficient training data. This resulted in:
  • Butterflies: 59 species, 127,671 specimens.
  • Moths: 283 species, 222,537 specimens.
  • Test Data: The remaining images formed the test set.

Key Insights and Ideas

  • RPV as a Confidence Score for Mislabelling: The Reoccurring Prediction Value (RPV) is a novel and effective metric. It moves beyond a single prediction to assess the consistency of misclassification, providing a more robust indicator of potential errors.
  • CV as a Triage System: The CV pipeline acts as a powerful first-pass filter, dramatically reducing the number of specimens requiring manual inspection. This is crucial for resource-constrained institutions.
  • Synergy is Key: The paper strongly advocates for a multi-faceted approach. CV alone isn't perfect, and genetic analysis is expensive. The combination of CV flagging, expert visual verification, and targeted genetic analysis offers the most accurate and efficient solution.
  • Human Error is the Dominant Factor: The study's findings suggest that the majority of mislabelling issues stem from human error during initial collection, curation, or digitization, rather than inherent flaws in the specimens themselves.
  • Portal Errors are Significant: Beyond mislabelling on physical specimen labels, errors in the digital data portal (e.g., incorrect species association or retrieval issues) are also a notable problem.

Key Results

  • Flagging Rate: Out of 350,208 total specimens (butterflies and moths), 99,350 (28.37%) were flagged by the pipeline at least once across 100 runs.
  • High RPV Specimens: While many specimens were flagged, a much smaller fraction had high RPVs. For butterflies, less than 1% had an RPV of 91-100. For moths, less than 0.12% had an RPV of 91-100. This indicates that only a small subset of flagged specimens are consistently misidentified by the model.
  • Expert Verification Success: Taxonomic experts visually examined 210 specimens (120 butterflies, 90 moths) with high RPVs (>80). They found that 145 (69%) were indeed incorrectly labelled in the collection or the NHM data portal.
  • Genetic Verification Confirmation: 30 specimens were sent for genetic verification. For butterflies, 15 out of 19 flagged specimens were confirmed as incorrectly identified by the pipeline. For moths, 3 out of 19 were confirmed. Notably, in some cases, genetic analysis contradicted both the pipeline and the visual experts.

Limitations and Future Work

  • Size and Location Data: The current CV model does not incorporate specimen size or geographical location data, which are crucial for distinguishing certain species. Initial attempts to include scale bars in training images led to the model focusing on them, so this remains a challenge.
  • Model Focus: While YOLOv8 cropping helped, the model can still be influenced by irrelevant background noise if not perfectly cropped.
  • Training Data Quality: The presence of mislabelled specimens in the training data can negatively impact model accuracy.
  • Future Work: The authors suggest incorporating contextual data like specimen size and location into future models. They also emphasize the need for user-friendly interfaces to allow museum staff to easily interact with and validate the CV pipeline's results. Further research into the relationship between RPV values and pipeline accuracy is also recommended.

Summary updated

Abstract

Natural history collections are essential for biodiversity and evolution research and for studying biotic responses to global change. However, the numbers of specimens within natural history collections pose management challenges. Reduced funds, declining taxonomic training and expanding collections can lead to mislabelled or missing specimens. This highlights the need for innovative and non-destructive methods of taxonomic verification for specimens in large collections. While genetic analyses offer precise verification, they are resource-intensive and less effective on degraded DNA from older specimens, with risks of damage to smaller specimens. Computer vision can automate tasks such as species-level verification and morphological examination, though these techniques have yet to be incorporated and utilised by natural history collections for such management tasks. Digitisation initiatives, such as those at the Natural History Museum (NHM), London, have gained momentum in recent years, converting specimens to digital formats and enhancing global accessibility. Here, we describe a computer vision pipeline applied to the digitised British and Irish Lepidoptera collection at the NHM. Specifically, our pipeline identifies specimens that do not match their labelled species status. The pipeline was executed for 100 runs for the Butterfly and Moth datasets, resulting in 99,350 out of 350,208 specimens (28.37%) being flagged at least once. We attribute a portion of these as pipeline errors, given the likelihood of some mislabelled specimens within training datasets. However, specimens flagged consistently across > 80% of pipeline runs are likely mislabelled within the collections. Taxonomic experts visually examined 210 such specimens, finding 145 to be incorrectly labelled in the collection or the NHM data portal. Additionally, 30 specimens were sent for genetic verification to confirm species-level identification. This synergy of computer vision and genetic-based species identification enhances the accuracy and efficiency of managing natural history collections, preserving their value for future generations.
BibTeX
@inproceedings{hollister2025computer,
  author = {Jack D. Hollister and G. Martin and Xiaohao Cai and Tammy Horton and Olivia Powell and Mark Sterling and Glory Turnbull and Benjamin Price and Phillip B. Fenberg},
  title = {{A Computer Vision Method for Finding Mislabelled Specimens Within Natural History Collections}},
  booktitle = {Ecology},
  year = {2025},
  doi = {10.1002/ece3.71648},
}
Figure