TL;DR
This paper presents systems for SemEval-2024 Task 7, focusing on numerical understanding and generation. The authors leverage instruction fine-tuning with large language models (LLMs) like Flan-T5 and T5, exploring techniques like demonstrations and chain-of-thought prompting to achieve strong results across quantitative understanding, Chinese reading comprehension, and numeral-aware headline generation.
What Problem Does This Paper Address?
Precise numerical information is crucial in many domains, yet current language models struggle to accurately understand and generate text containing numbers. This can lead to significant inaccuracies in decision-making and planning. SemEval-2024 Task 7 was designed to address this gap by focusing on three key areas: quantitative understanding (English), reading comprehension of numerals in text (Chinese), and numeral-aware headline generation (English). The challenge lies in developing models that can reliably process, interpret, and generate numerical data within natural language contexts.
Methods
The core of the proposed approach is instruction fine-tuning applied to pre-trained LLMs such as Flan-T5 and T5. The authors explore two main strategies for instruction tuning:
- Manual-based Instruction Tuning: This involves crafting specific instruction templates and, for some tasks, manually selecting demonstrations (examples of input-output pairs) to guide the model. This was applied to Subtask 1.
- Similarity-based Instruction Tuning: For Subtask 3 (headline generation), they employed a more efficient method where similar news articles were retrieved using Sentence-BERT embeddings to construct demonstrations.
A key innovation for improving reasoning performance, particularly in Subtask 3's numerical reasoning, is the use of Chain-of-Thought (CoT) prompting. Instead of directly outputting a numerical answer, the model's output is reformulated into a step-by-step reasoning process, including calculation expressions. This makes the model's reasoning more interpretable and can be further enhanced by using an external calculator for result correction.
Datasets
The paper utilizes several datasets for training and evaluation:
- Quantitative 101 (Chen et al., 2023): Used for Subtask 1, this dataset covers three aspects: QP (Question Answering), QQA (Question Answering), and QNLI (Natural Language Inference).
- NQuAD (Chen et al., 2021): A Chinese machine reading comprehension dataset used for Subtask 2.
- NumHG (Huang et al., 2023): This dataset, comprising over 27,000 annotated numeral-rich news articles, is used for Subtask 3, encompassing both headline generation and numerical reasoning.
Key Insights and Ideas
- Instruction Tuning is Powerful: The paper strongly supports the effectiveness of instruction fine-tuning for numerical tasks, showing that well-crafted instructions significantly improve LLM performance compared to direct fine-tuning.
- CoT for Reasoning: Reformulating numerical reasoning outputs into a Chain-of-Thought format, especially when combined with an external calculator for verification, dramatically boosts performance on complex reasoning tasks. This highlights the importance of explicit reasoning steps for LLMs.
- Demonstrations Matter, but How Matters More: While demonstrations generally help, the paper notes that manual selection for Subtask 1 didn't yield significant improvements, suggesting that the quality and relevance of demonstrations (as seen in the similarity-based approach for Subtask 3) are more critical than simply having them.
- Original Notation Beats Digit-Based: For quantitative understanding, using the original numerical notation in text performed better than segmenting numbers at the character level, indicating that preserving the inherent numerical structure is beneficial.
- Larger Models Generally Perform Better: Across various subtasks, especially headline generation and numerical reasoning, larger Flan-T5 models (XL and XXL) consistently outperformed smaller ones, underscoring the scalability of the proposed methods.
Key Results
The systems achieved competitive results across all subtasks:
- Subtask 1 (Quantitative Understanding): The instruction-tuned Flan-T5 model performed comparably to BERT, with notable superiority on QNLI and QQA.
- Subtask 2 (Chinese Reading Comprehension): T5-based models (mT5 and Randeng-T5) outperformed BERT, with Randeng-T5 showing an edge due to its Chinese corpus pre-training.
- Subtask 3 (Numeral-Aware Headline Generation): The system achieved the highest scores for ROUGE, BERTScore, and MoverScore.
- Subtask 3 (Numerical Reasoning): The system ranked 3rd overall. The CoT method, especially with external calculator correction, significantly improved performance on complex reasoning questions.
Limitations and Future Work
- Manual Demonstration Limitations: The paper acknowledges that manual selection of demonstrations in Subtask 1 did not lead to expected improvements, possibly due to issues with example selection or hyperparameter tuning.
- Model Scale Impact: While larger models generally perform better, the paper notes that for Subtask 2, increasing model scale seemed to lead to decreased accuracy, suggesting a potential need for task-specific tuning strategies for very large models.
- Future Work: The authors plan to further investigate the impact of varying:
- Instance quantities in demonstrations.
- Instruction templates.
- Model sizes.
Summary updated