Machine Learning Using Digitized Herbarium Specimens to Advance Phenological Research

Katelin D. Pearson, Gil Nelson, Myla F. J. Aronson, Pierre Bonnet, Laura Brenskelle, Charles C. Davis, Ellen G. Denny, Elizabeth R. Ellwood, Hervé Goëau, J. Mason Heberling, Alexis Joly, Titouan Lorieul, Susan J. Mazer, Emily K. Meineke, Brian J. Stucky, Patrick W. Sweeney, Alexander E. White, Pamela S. SoltisBioScience 2020 · DOI

TL;DR

This paper introduces a generalized machine learning (ML) workflow to automatically extract plant phenological data from digitized herbarium specimen images. This approach promises to overcome the significant bottleneck of manual annotation, enabling large-scale phenological research crucial for understanding climate change impacts on ecosystems.

What Problem Does This Paper Address?

Plant phenology, the timing of life cycle events like flowering and fruiting, is a critical indicator of climate change and a driver of ecological processes. Understanding these patterns requires vast amounts of historical data, which have traditionally been collected through laborious manual annotation of herbarium specimens. This manual process is time-consuming, expensive, and limits the scale and scope of phenological research. The paper addresses the challenge of efficiently and accurately extracting phenological information from millions of digitized herbarium specimens to enable broader and deeper ecological and evolutionary studies.

Methods

The authors propose a generalized, modular machine learning workflow for phenological annotation of herbarium specimen images. The core of this workflow involves:

  • Storage: Managing digitized herbarium specimen images, associated metadata, and ML predictions.
  • Generating Training Data: Manually annotating a subset of specimen images with phenological information (e.g., presence/absence of flowers, bounding boxes around reproductive structures). This is a crucial step, as ML models learn from these annotated examples.
  • Phenophase Annotation: This component involves using ML models to detect and quantify reproductive structures. Techniques like Convolutional Neural Networks (CNNs), and specifically object detection (e.g., R-CNN) and instance segmentation (e.g., Mask R-CNN), are highlighted as effective for this task.
  • Machine Learning (ML) / Deep Learning: Training ML models, particularly deep learning CNNs, on the annotated training data. The process involves model selection, training, and validation.
  • Testing/Analysis: Evaluating the trained ML model's performance using a separate set of manually annotated data to measure accuracy and precision.
  • Deployment: Applying the trained model to predict phenological annotations for previously unannotated specimens. This can be done via web services, standalone programs, or GUIs.

The paper emphasizes the use of deep learning, specifically CNNs, due to their ability to automatically learn relevant features from images, making them well-suited for complex visual tasks like identifying reproductive structures.

Datasets

The paper does not introduce a new benchmark dataset. Instead, it discusses the use of digitized herbarium specimens as the primary data source. These specimens are preserved plant material curated in natural history collections. The authors highlight that millions of specimen records are already digitized and available through platforms like iDigBio. The workflow relies on these existing digitized images and the core specimen data (collection date, scientific name, locality) that are often captured during digitization. For training and validation, a subset of these digitized images is manually annotated.

Key Insights and Ideas

  • Herbarium specimens are a goldmine for phenological data: Digitized herbarium collections represent a vast, underutilized resource for studying long-term phenological changes, offering unparalleled spatio-temporal coverage.
  • ML can automate a critical bottleneck: Manual annotation of herbarium specimens is a major hurdle. ML, particularly deep learning, offers a scalable solution to automate this process, enabling research at unprecedented scales.
  • Deep learning excels at visual tasks: CNNs are particularly effective for image analysis tasks like object detection and segmentation, making them ideal for identifying and quantifying reproductive structures on specimens.
  • Training data quality is paramount: The accuracy and generalizability of ML models are heavily dependent on the quality and diversity of the manually annotated training data. Biases in training data will be reflected in model predictions.
  • ML can go beyond simple presence/absence: While early phenological studies focused on basic events, ML can quantify more nuanced traits like flowering duration, progression rates, and the number of reproductive structures, leading to a richer understanding.

Key Results

The paper focuses on presenting a workflow and discussing its potential rather than reporting specific quantitative results from a single experiment. However, it cites previous work and makes strong claims about the potential of ML:

  • High Accuracy: The deep learning method evaluated in a cited study (Lorieul et al. 2019) achieved an accuracy of 96.3% for detecting fertile specimens, outperforming human experts in some cases. Accuracy for finer-scale annotations (flowers, fruits) was slightly lower but still high (e.g., 84.3% for flower detection).
  • Speed: Once trained, ML algorithms can score tens to thousands of specimens per minute with a single GPU, a dramatic improvement over manual annotation.
  • Scalability: The workflow is designed to handle millions of images, enabling phenological studies at scales previously unimaginable.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Training Data Requirements: The need for extensive, visually diverse, and accurately annotated training data remains a significant challenge. Continued digitization and innovative annotation strategies (e.g., citizen science) are crucial.
  • Specimen Quality and Condition: Herbarium specimens themselves can limit ML utility due to damaged or obscured structures, or the absence of key phenological indicators. ML cannot fully infer phenology if the necessary structures are not present or visible.
  • Non-plant Material: Labels, stamps, and other mounting materials on specimens can introduce noise and bias into ML models.
  • Generalizability: Developing models that generalize across diverse taxa, geographic regions, and specimen preparation techniques is an ongoing challenge.
  • Cyberinfrastructure: Advanced computing resources (GPUs) and collaborative platforms are needed to support large-scale ML experiments.
  • Interoperability: Ensuring that ML-generated data is compatible with existing biodiversity data standards (e.g., Darwin Core, Plant Phenology Ontology) is essential for its downstream use.
  • Diversifying Research: Future work should extend ML applications beyond basic flowering phenology to include vegetative phenology, less-studied taxa, and species interactions.
  • Collaboration: Increased collaboration between biologists and computer scientists, as well as with industry partners, is vital for advancing the field.

Summary updated

Abstract

Machine learning (ML) has great potential to drive scientific discovery by harvesting data from images of herbarium specimens-preserved plant material curated in natural history collections-but ML techniques have only recently been applied to this rich resource. ML has particularly strong prospects for the study of plant phenological events such as growth and reproduction. As a major indicator of climate change, driver of ecological processes, and critical determinant of plant fitness, plant phenology is an important frontier for the application of ML techniques for science and society. In the present article, we describe a generalized, modular ML workflow for extracting phenological data from images of herbarium specimens, and we discuss the advantages, limitations, and potential future improvements of this workflow. Strategic research and investment in specimen-based ML methods, along with the aggregation of herbarium specimen data, may give rise to a better understanding of life on Earth.
BibTeX
@inproceedings{pearson2020machine,
  author = {Katelin D. Pearson and Gil Nelson and Myla F. J. Aronson and Pierre Bonnet and Laura Brenskelle and Charles C. Davis and Ellen G. Denny and Elizabeth R. Ellwood and Hervé Goëau and J. Mason Heberling and Alexis Joly and Titouan Lorieul and Susan J. Mazer and Emily K. Meineke and Brian J. Stucky and Patrick W. Sweeney and Alexander E. White and Pamela S. Soltis},
  title = {{Machine Learning Using Digitized Herbarium Specimens to Advance Phenological Research}},
  booktitle = {BioScience},
  year = {2020},
  doi = {10.1093/biosci/biaa044},
}
Figure