Evaluating Named-Entity Recognition approaches in plant molecular biology

Huy Do, Khoat Than, Pierre Larmande

TL;DR

This paper tackles the challenge of extracting crucial biological information like gene and protein names from scientific literature in plant molecular biology, specifically for rice. The authors introduce a new benchmark dataset, Oryzabase, and evaluate various Named Entity Recognition (NER) approaches, finding that a Long Short-Term Memory (LSTM) model combined with Conditional Random Fields (CRFs) achieves state-of-the-art performance, reaching an impressive 86.72% F1-score.

What Problem Does This Paper Address?

The explosion of information in life sciences, particularly in fields like agronomy and plant molecular biology, presents a significant challenge. Much of this valuable knowledge is locked away in unstructured scientific documents like journal articles and reports, rather than being readily available in structured databases. For plant molecular biology, and specifically for rice research, there's a notable lack of comprehensive studies and readily available datasets for training advanced machine learning models to extract key biological entities such as genes, proteins, and phenotypic traits. This deficit hinders the development of applications that could accelerate crop improvement and deepen our understanding of plant biology.

Methods

The paper explores and evaluates several approaches for Named Entity Recognition (NER) in plant molecular biology text. The core methods investigated are:

  • LSTM-CRF: This is a deep learning approach that combines a Long Short-Term Memory (LSTM) network with Conditional Random Fields (CRFs). LSTMs are adept at handling sequential data and capturing long-term dependencies in text, while CRFs are effective for sequence labeling tasks. The authors use a bidirectional LSTM to capture context from both directions of a sentence, and then feed this into a CRF layer for final entity tagging. This is presented as a sophisticated, state-of-the-art method.
  • Hybrid Approach (OGER): This method combines dictionary lookup with machine learning. OGER (Onto Gene group's entity recognizer) uses a dictionary-based approach to identify entities. This output is then used as features for machine learning algorithms, specifically Conditional Random Fields (CRFs) and Neural Networks (NNs), to refine the entity recognition.

Datasets

The primary dataset used in this study is Oryzabase, a new benchmark dataset introduced by the authors. It's derived from the Oryzabase scientific database for Oryza sativa (rice) and includes abstracts and titles from over 10,000 scientific articles. The dataset was curated by matching keywords against the Oryzabase gene list to identify gene entities. The data was split into 75% for training, 15% for testing, and 10% for validation to prevent overfitting.

Key Insights and Ideas

  • The Need for Specialized Benchmarks: The paper highlights that existing benchmarks are insufficient for plant molecular biology, especially for rice. The creation of Oryzabase is a crucial step towards enabling more robust research in this area.
  • LSTM-CRF as a Powerful Tool: The study strongly suggests that advanced deep learning architectures like LSTM-CRF are highly effective for extracting biological entities from complex scientific text, outperforming simpler methods.
  • Hybrid Approaches Offer Promise: While LSTM-CRF leads, hybrid methods combining dictionary lookup with machine learning also show competitive results, indicating that leveraging existing knowledge bases can be a valuable strategy.
  • Data Imbalance is a Challenge: The authors encountered issues with imbalanced entity types in the dataset, which negatively impacted initial model performance, underscoring the importance of addressing data distribution in NER tasks.

Key Results

The paper presents a comparative analysis of the evaluated models:

  • LSTM-CRF: Achieved the best overall performance with an average F1-score of 86.72%.
  • Pure LSTM: Performed reasonably well with an average F1-score of 80.44%.
  • Hybrid (OGER + CRF): Showed strong results, achieving an average F1-score of 85.08% on the testing set.
  • Hybrid (OGER + NN): Reached an average accuracy of 67%.
  • OGER (dictionary lookup only): Achieved around 58.5%, demonstrating the significant improvement gained by integrating machine learning.

Limitations and Future Work

Limitations:

  • Data Imbalance: The dataset exhibited imbalances in entity types, which posed a challenge during training and required specific strategies to mitigate.
  • Pre-processing Effort: The pre-processing of the large dataset, including tokenization and entity tagging, was time-consuming.

Future Work:

  • Optimizing Accuracy and Speed: The authors plan to explore further optimizations to improve both accuracy and computational efficiency.
  • High-Performance Computing (HPC): They intend to leverage GPU-accelerated HPC for training deep sequential models on large datasets, aiming to significantly reduce computing time.
  • Parallel Computing for Sequential Models: A key challenge identified for future work is effectively applying parallel computing techniques to sequential models.

Summary updated

Abstract

Abstract Text mining research is becoming an important topic in biology with the aim to extract biological entities from scientific papers in order to extend the biological knowledge. However, few thorough studies on text mining and applications are developed for plant molecular biology data, especially rice, thus resulting a lack of datasets available to train models able to detect entities such as genes, proteins and phenotypic traits. Since there is rare benchmarks for rice, we have to face various difficulties in exploiting advanced machine learning methods for accurate analysis of rice bibliography. In this article, we developed a new training datasets (Oryzabase) as the benchmark. Then, we evaluated the performance of several current approaches to find a methodology with the best results and assigned it as the state of the art method for our own technique in the future. We applied Name Entities Recognition (NER) tagger, which is built from a Long Short Term Memory (LSTM) model, and combined with Conditional Random Fields (CRFs) to extract information of rice genes and proteins. We analyzed the performance of LSTM-CRF when applying to the Oryzabase dataset and improved the results up to 86% in F 1 . We found that on average, the result from LSTM-CRF is more exploitable with the new benchmark.
BibTeX
@inproceedings{do2018evaluating,
  author = {Huy Do and Khoat Than and Pierre Larmande},
  title = {{Evaluating Named-Entity Recognition approaches in plant molecular biology}},
  booktitle = {MIWAI},
  year = {2018},
  doi = {10.1007/978-3-030-03014-8\_19},
}
Figure