CYUT at SemEval-2024 Task 7: A Numerals Augmentation and Feature Enhancement Approach to Numeral Reading Comprehension

Tsz-yeung Lau, Shih-hung WuNAACL 2024

TL;DR

This paper tackles the challenge of numeral reading comprehension in Chinese news articles by proposing a novel approach that augments numeral data and enhances feature representation within a BERT-based model. Their method, BERT-NAFE, significantly improves accuracy by addressing numeral ambiguity and reducing model over-reliance on specific numerical values, achieving a notable 7.14% increase over the baseline.

What Problem Does This Paper Address?

Reading comprehension, especially concerning numerical information, is a critical but challenging task in natural language processing. While numbers are ubiquitous and crucial in domains like finance, medicine, and engineering, machine comprehension of these numerical entities remains an emerging research area. SemEval-2024 Task 7 specifically focuses on numeral reading comprehension in Chinese news articles, requiring models to identify the correct numerical value from a set of options based on a given article. The existing dataset, NQuAD, presents a complex challenge due to the inherent ambiguity and context-dependency of numerals. The authors aim to improve upon existing baseline models by developing a more robust approach to numeral understanding.

Methods

The core of this work lies in enhancing a pre-trained BERT model through a series of carefully designed preprocessing and augmentation techniques.

  • BERT Baseline: The study starts with a standard pre-trained BERT model, a powerful transformer-based architecture known for its strong performance in various NLP tasks.
  • Stop Word Removal (BERT-SWR): Common, uninformative words (stop words) are removed from the text. This step aims to reduce noise, decrease the dimensionality of the feature space, and allow the model to focus on more semantically rich content.
  • Numerals Augmentation (NA) and Feature Enhancement (FE) (BERT-NAFE): This is the paper's main contribution.
  • Numerals Augmentation: Numerical values in the answer options are replaced with their corresponding indices (0, 1, 2, 3). This technique is designed to reduce the model's over-reliance on specific numerical values and promote a more generalized understanding of numerical information across different contexts.
  • Feature Enhancement: Special symbols (cashtags) are added before and after the replaced numerical values in the text. This is intended to draw the model's attention mechanism towards these markers, thereby increasing its focus on processing the surrounding content and improving its ability to recognize and interpret numerical information.

Fine-tuning was applied to all models to adapt them to the specific task.

Datasets

The primary dataset used in this study is the Numeral-related Question Answering Dataset (NQuAD). This dataset is specifically designed for numeral reading comprehension in Chinese news articles. It comprises 75,448 Chinese news articles spanning from June 2013 to June 2018, with a high prevalence of numerals in both headlines (59.74%) and content (99.80%). The dataset contains 43,787 news articles and 71,998 questions, structured into six columns: "news_article", "question_stem", "answer_options", "ans", "target_num", and "sentences_containing_the_numeral_in_answer_options". The authors note an approximate 8:2 ratio for the training and test sets, with 57,750 samples in the training set and 14,438 in the test set.

Key Insights and Ideas

  • The Power of Noise Reduction: Simply removing stop words from the text led to a significant improvement in accuracy, highlighting how much "noise" these common words can introduce for a model trying to grasp core meaning.
  • Decoupling Numerals from Specific Values: Replacing numerical answers with indices (0, 1, 2, 3) is a clever way to force the model to learn the relationship between the question and the answer's position, rather than just memorizing specific numbers. This makes the model more robust to variations in numerical values.
  • Attention is All You Need (with Markers): The use of cashtags as feature enhancement is a simple yet effective way to guide the model's attention mechanism. By explicitly marking the areas of interest, the model is nudged to focus on the contextual clues surrounding numerals, which is crucial for disambiguation.
  • Combined Approach Yields Best Results: The most significant gains were achieved when combining stop word removal with numeral augmentation and feature enhancement, suggesting that a multi-faceted approach is key to tackling complex numeral comprehension tasks.

Key Results

The study reports the following performance improvements:

  • Initial NEMo Model (Chen et al., 2021): 69.95% accuracy.
  • BERT Baseline (fine-tuned): 67.95% accuracy (a slight decrease from NEMo).
  • BERT-SWR (Stop Word Removal): 70.83% accuracy (a marginal improvement of ~1% over NEMo).
  • BERT-NAFE (Numerals Augmentation and Feature Enhancement): 77.09% accuracy. This represents a substantial 7.14% increase compared to the initial NEMo model and a 9.14% increase over the BERT baseline.

The authors also note that the model achieved satisfactory performance within just 2 epochs of fine-tuning.

Limitations and Future Work

  • Data Distribution: The confusion matrix analysis revealed that the system tends to over-predict class 2 outputs and under-predict class 3 outputs. The authors suggest that adjusting the training set distribution to a balanced 1:1:1:1 ratio could improve performance. They also note that the numerical labels (0, 1, 2, 3) are arbitrary and their distribution can be easily adjusted.
  • "Scao" Column Limitations: In some cases, the "sentences_containing_the_numeral_in_answer_options" (scao) column does not contain the answer to the question, forcing the model to predict based on estimation. The authors propose using the full "news_article" as a substitute, but acknowledge this would increase token length and computational costs.
  • Symbol Experimentation: While various symbols were tested for feature enhancement, comparable performance was observed. However, using "triple symbols" led to a slight decrease in accuracy. Future work could explore more sophisticated symbol choices or alternative feature enhancement strategies.
  • Computational Cost: The proposed substitution of the "scao" column with the full "news_article" would increase token length, leading to higher training costs (GPU memory and time). This trade-off needs to be considered for future applications.

Summary updated

BibTeX
@inproceedings{lau2024cyut,
  author = {Tsz-yeung Lau and Shih-hung Wu},
  title = {{CYUT at SemEval-2024 Task 7: A Numerals Augmentation and Feature Enhancement Approach to Numeral Reading Comprehension}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure