TL;DR
The HIJLI_JU team tackled SemEval-2024 Task 7, focusing on quantitative question answering (QQA) by fine-tuning a BERT model. Their approach leverages the Hugging Face ecosystem to enhance numerical comprehension in multiple-choice questions, demonstrating that character-based numerical representations yield the best performance.
What Problem Does This Paper Address?
Quantitative Question Answering (QQA) is crucial for extracting meaningful insights from large datasets across various industries like finance and healthcare. However, existing AI systems, including large language models, often struggle with arithmetic reasoning and understanding numerical information presented in diverse formats. This paper addresses the challenge of improving a model's ability to accurately answer quantitative questions, especially when numerical values are represented in different ways (e.g., standard numbers, scientific notation, or spelled-out characters).
Methods
The HIJLI_JU team employed a fine-tuned BERT model for their QQA task. They utilized Hugging Face's BertTokenizer and TFBertForMultipleChoice for training and inference. The data preparation involved adding "context" and "label" to JSON files, transforming them into Dataset objects, and using a preprocess_function with the map method. To optimize performance, they implemented dynamic sentence padding during collation using a DataCollatorForMultipleChoice modification. The core of their approach lies in fine-tuning a pre-trained BERT model, known for its bidirectional capabilities in understanding semantic nuances.
Datasets
The paper utilizes the SemEval-2024 Task 7 dataset, which is part of the NumEval Task 1: Quantitative Understanding benchmark. This dataset includes three task types: Quantitative Prediction (QP), Quantitative Natural Language Inference (QNLI), and Quantitative Question Answering (QQA). The QQA task specifically involves questions with options, correct answers, and various attribute fields that represent numerical values in different formats:
question: Standard numerical representation.question_sci_10E: Numerical values in scientific notation.question_char: Numerical values as characters (e.g., "one hundred").question_sci_10E_char: A combination of scientific notation and characters.question_mask: Questions with placeholders like "[Num]".
The dataset statistics are provided: 564 examples for training, 81 for development, and 162 for testing.
Key Insights and Ideas
- Character-based representation excels: The most significant finding is that questions where numerical values are represented as characters (
question_char) consistently outperform other formats. This suggests that models might be better at processing and reasoning with spelled-out numbers compared to standard digits or scientific notation in this specific QQA task. - Robustness across formats: The
question_charformat demonstrates robustness in handling diverse classes independently, even in imbalanced datasets, and contributes proportionally to the overall performance. - BERT's suitability for QQA: The paper reinforces the effectiveness of fine-tuned BERT models for quantitative reasoning tasks, highlighting their ability to capture semantic subtleties crucial for answering numerical questions.
Key Results
The paper reports F1 scores across various metrics and question formats. The question_char format consistently achieved the highest Macro-F1, Micro-F1, and Weighted-F1 scores. For instance:
question_charachieved a Macro-F1 of 0.53058, Micro-F1 of 0.53704, and Weighted F1 of 0.53058.- In comparison, the standard
questionformat yielded a Macro-F1 of 0.50344, Micro-F1 of 0.50617, and Weighted F1 of 0.50345. - The
question_sci_10Eformat performed the lowest, with a Macro-F1 of 0.44026. - The
question_sci_10E_charformat showed strong performance, close toquestion_char, with a Macro-F1 of 0.51489.
The Binary F1 scores also indicate that the question_char format leads to better performance in distinguishing between correct and incorrect answers.
Limitations and Future Work
Limitations:
* The paper acknowledges that the overall F1 scores are moderate, suggesting room for improvement in the model's quantitative reasoning capabilities.
* While the question_char format performed best, the absolute scores indicate that the model still struggles with certain aspects of quantitative understanding.
Future Work: * The authors suggest exploring new QQA tasks and applying them to specific domains like healthcare. * They propose collaborating with experts in different fields to enhance the utility of QQA. * Future research could focus on improving evaluation metrics and leveraging the latest advancements in language understanding technologies to further refine QQA models.
Summary updated