Stanford MLab at SemEval-2021 Task 8: 48 Hours Is All You Need

Patrick Liu, Niveditha Iyer, Erik Rozi, Ethan A. ChiACL 2021

TL;DR

This paper presents a surprisingly effective BERT-based system for extracting quantities, units, and modifiers from scientific text, developed in just 48 hours. Despite its rapid development and lightweight nature, the system achieved a strong 12th place overall and 2nd place for unit identification in the SemEval-2021 MeasEval task, demonstrating the power of pre-trained models for complex NLP challenges.

What Problem Does This Paper Address?

Analyzing scientific literature is becoming increasingly difficult as the volume of research grows. While natural language processing (NLP) has made strides in tasks like text summarization, extracting specific, structured information like measurements and their associated units and modifiers from scientific text remains a significant challenge. This information is crucial for tasks like automated data analysis and knowledge discovery, but its extraction is hampered by inconsistent formatting and varied placement within sentences. This paper tackles the problem of automatically identifying and classifying these measurement-related components within scientific documents.

Methods

The authors leveraged pre-trained BERT (Bidirectional Encoder Representations from Transformers) models, a powerful architecture for understanding language context. For the core tasks of identifying "Quantity" spans (e.g., "13 kg") and "Unit" spans (e.g., "kg"), they fine-tuned BERT-Large using a token-level classification approach. Specifically, they employed an "IO" labeling strategy, where each token is either inside ("I") or outside ("O") a target span. This is a common technique in Named Entity Recognition (NER).

For classifying "Modifiers" (e.g., indicating a range or mean value), they treated it as a multilabel classification problem. Each quantity could have multiple modifiers, so they mapped each modifier to an individual class. A BERT model was used to tokenize the quantity, and the output was fed into a simple linear layer to predict the likelihood of each modifier being present.

A key aspect of their approach was its rapid implementation, achieved in approximately 48 hours by authors with limited prior formal NLP experience. They prioritized using readily available tools like the Hugging Face Transformers library and BERT's built-in BERTForTokenClassification class to expedite development.

Datasets

The paper used data provided for the SemEval-2021 Task 8: MeasEval. This dataset consists of scientific text paragraphs annotated with quantities, units, and modifiers. The authors utilized a train-trial split of 2531 annotations across 249 texts for training and 832 annotations across 66 texts for evaluation.

Key Insights and Ideas

  • Rapid Development is Feasible: The paper strongly demonstrates that even with significant time constraints (48 hours) and limited prior experience, a robust and competitive NLP system can be built using pre-trained models.
  • BERT's Versatility: BERT proved effective across different subtasks, from span identification to multilabel classification, highlighting its adaptability for various NLP challenges.
  • Simplicity Can Be Effective: The authors deliberately chose simpler implementations (e.g., a single linear layer for modifier classification, avoiding complex multi-layer fine-tuning) to save time, and these choices did not significantly hinder performance.
  • Post-processing Matters: Simple post-processing steps, like removing trailing punctuation and discarding non-numeric quantities, significantly improved precision, indicating that the base model tended to overpredict.

Key Results

The system achieved a final overall F1 score of 0.272, ranking 12th out of 19 teams in the evaluation phase. Notably, it secured 2nd place in the "Unit" identification subtask with an F1 score of 0.76. The "Quantity" identification subtask also performed well, with an F1 score of 0.818, ranking 10th. The "Modifier" classification subtask achieved an F1 score of 0.408, ranking 8th. The paper also showed that post-processing improved the precision of quantity identification from 0.564 to 0.8153.

Limitations and Future Work

The authors acknowledge that their system was developed under significant time pressure, which may have limited the extent of hyperparameter tuning and architectural exploration. They suggest that future work could involve exploring more powerful models beyond BERT and augmenting the training data, particularly in areas where the model frequently makes erroneous predictions.

Summary updated

BibTeX
@inproceedings{liu2021stanford,
  author = {Patrick Liu and Niveditha Iyer and Erik Rozi and Ethan A. Chi},
  title = {{Stanford MLab at SemEval-2021 Task 8: 48 Hours Is All You Need}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure