PublishInCovid19 at WNUT 2020 Shared Task-1: Entity Recognition in Wet Lab Protocols using Structured Learning Ensemble and Contextualised Embeddings

Janvijay Singh, Anshul Wadhawan

Abstract

In this paper, we describe the approach that we employed to address the task of Entity Recognition over Wet Lab Protocols -- a shared task in EMNLP WNUT-2020 Workshop. Our approach is composed of two phases. In the first phase, we experiment with various contextualised word embeddings (like Flair, BERT-based) and a BiLSTM-CRF model to arrive at the best-performing architecture. In the second phase, we create an ensemble composed of eleven BiLSTM-CRF models. The individual models are trained on random train-validation splits of the complete dataset. Here, we also experiment with different output merging schemes, including Majority Voting and Structured Learning Ensembling (SLE). Our final submission achieved a micro F1-score of 0.8175 and 0.7757 for the partial and exact match of the entity spans, respectively. We were ranked first and second, in terms of partial and exact match, respectively.

Introduction

Reproducibility is a major hurdle in biological research. While "wet lab" protocols—the step-by-step instructions for experiments—are essential, they are mostly written in natural language filled with jargon, colloquialisms, and inconsistent formatting. This makes it incredibly difficult for machines to "read" a protocol and translate it into executable actions for robotic automation.

To bridge this gap, we first need to solve Entity Recognition (ER): the ability to automatically identify and categorize specific terms in a protocol, such as reagents, temperatures, locations, and actions. The challenge lies in the high density of technical terms and the fact that many words (like "1.5 ml") could represent an Amount, a Size, or a Volume depending entirely on the context.

Methodology

The authors tackled the problem in two distinct phases: architecture optimization and ensembling.

Architecture Optimization

The researchers experimented with various contextualized word embeddings, finding that general-purpose models (like BERT-base) lagged behind domain-specific ones. The best-performing setup used a concatenation of PubMedBERT (trained on biomedical text) and Flair (a character-level embedding). These were fed into a BiLSTM-CRF (Bidirectional Long Short-Term Memory with Conditional Random Fields) model. Combining Transformer-based embeddings with character-based embeddings allowed the model to capture both high-level semantic meaning and low-level morphological features.

Performance comparison of various base models and architectures, showing the superiority of the PubMedBERT + Flair combination.

Structured Learning Ensemble (SLE)

Instead of relying on a single model, the team trained 11 different versions of their architecture on random splits of the data. To combine their predictions, they compared standard Majority Voting against Structured Learning Ensembling (SLE). While majority voting just picks the most common label for each word, SLE builds transition matrices to understand the "grammar" of the labels (e.g., an "Intermediate" tag should logically follow a "Begin" tag). This global optimization ensures the final output is structurally coherent.

Key Results and Insights

The "PublishInCovid19" system was highly successful at the WNUT 2020 Shared Task, achieving 1st Place in Partial Match (Micro F1-score: 0.8175) and 2nd Place in Exact Match (Micro F1-score: 0.7757).

Key insights from the study include:

  • Domain Specificity is Non-Negotiable: Using PubMedBERT provided a significant boost over standard BERT, proving that pre-training on scientific literature is crucial for niche tasks like lab protocol analysis.
  • Structure Matters in Ensembles: The move from Majority Voting to SLE was a key breakthrough. By treating the ensemble output as a sequence-level optimization problem, the authors reduced logical errors in the predicted entity spans.

Error Analysis and Limitations

Despite the high performance, several persistent hurdles remain. The model frequently confused "Begin" and "Intermediate" tags for reagents it hadn't seen during training. Furthermore, the Modifier entity type is highly dependent on the entity it modifies, leading to occasional confusion with "Other" (O) text.

Error analysis table highlighting common misclassifications, particularly between Modifiers, Reagents, and the 'Other' (O) label.

Numerical ambiguity also remains a challenge; distinguishing between Concentration, Amount, and Size is difficult because these entities often share identical formats. For future work, the authors suggest integrating rule-based approaches to complement the neural network, specifically to handle these tricky numerical categories and improve the precision of reagent identification.

Summary updated

Abstract

In this paper, we describe the approach that we employed to address the task of Entity Recognition over Wet Lab Protocols -- a shared task in EMNLP WNUT-2020 Workshop. Our approach is composed of two phases. In the first phase, we experiment with various contextualised word embeddings (like Flair, BERT-based) and a BiLSTM-CRF model to arrive at the best-performing architecture. In the second phase, we create an ensemble composed of eleven BiLSTM-CRF models. The individual models are trained on random train-validation splits of the complete dataset. Here, we also experiment with different output merging schemes, including Majority Voting and Structured Learning Ensembling (SLE). Our final submission achieved a micro F1-score of 0.8175 and 0.7757 for the partial and exact match of the entity spans, respectively. We were ranked first and second, in terms of partial and exact match, respectively.
BibTeX
@inproceedings{singh2020publishincovid,
  author = {Janvijay Singh and Anshul Wadhawan},
  title = {{PublishInCovid19 at WNUT 2020 Shared Task-1: Entity Recognition in Wet Lab Protocols using Structured Learning Ensemble and Contextualised Embeddings}},
  booktitle = {W-NUT @ EMNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=vk45Yjn6Ux},
}
Figure