Abstract
Numerical reasoning remains a significant challenge for Large Language Models (LLMs), particularly in context-heavy domains like news reporting. This paper describes the approach taken by team ZXQ for SemEval-2024 Task 7 (NumEval), focusing on Subtask 3: deducing missing numbers in news headlines. By fine-tuning GPT-3.5-Turbo, the team achieved a high accuracy of 0.94, securing second place in the competition. Notably, the study finds that direct answer prediction outperforms reasoning-aware approaches that utilize intermediate symbolic steps.
Introduction
While LLMs demonstrate remarkable capabilities in creative writing and coding, precise arithmetic within news contexts is often prone to "hallucinations." News headlines frequently omit specific figures present in the main text to maintain brevity. Automated systems must accurately reconstruct these missing numerals to ensure factual consistency. This study explores the efficacy of fine-tuning proprietary LLMs to bridge this gap in numerical reasoning.
Task Definition and Dataset
The research addresses Numerical Reasoning (Subtask 3 of NumEval). Given a news article and a "masked" headline where a number has been removed, the system must predict the missing numeral. This requires the model to understand the text and perform operations like copying, rounding, or basic arithmetic based on the context provided.
The team utilized the NumHG dataset, which consists of 21,157 news articles. Each instance includes a masked headline and meticulous annotations describing how the missing number is derived from the text. The dataset identifies various fundamental operations required for reasoning, ranging from simple extraction to complex arithmetic.
The data was partitioned into an 80/20 split for training and testing, respectively.
Methodology
The core of the approach involved fine-tuning GPT-3.5-Turbo-0613 via the OpenAI API. The researchers implemented a structured pipeline involving system prompts, user inputs (news article and masked headline), and assistant outputs.
Two primary experimental configurations were tested:
- Direct Prediction (Experiment 1): The model was prompted to output only the final missing number directly from the news content.
- Reasoning-Aware Prediction (Experiment 2): Inspired by instruction tuning, the model was trained to generate intermediate symbolic operations (e.g.,
Add(v0, v1)) followed by the final answer.
A post-processing "data type validator" was employed during inference to ensure outputs were strictly numerical, converting any non-numeric failures to zero.
Results and Analysis
The Direct Prediction model (Experiment 1) emerged as the superior approach, achieving an accuracy of 0.94 and placing 2nd on the official SemEval-2024 leaderboard. In contrast, the Reasoning-Aware model (Experiment 2) achieved a lower accuracy of 0.90.
The "Simplicity" Paradox
Contrary to common findings where "Chain-of-Thought" reasoning improves LLM performance, this task saw a decline in accuracy when symbolic reasoning was required. This suggests that forcing the model into a rigid symbolic format may interfere with its natural predictive capabilities or internal heuristics.
Performance by Operation Type
The model's success was largely driven by its proficiency in "copying" tasks, which constitute 88.5% of the test data. However, performance degraded significantly on tasks requiring actual mathematical computation:
- Copy/Trans: Error rates were low (3.13% - 4.32%).
- Multiplication: Error rate of 35.71%.
- Rounding: Error rate of 41.08%.
These results indicate that while fine-tuned LLMs are excellent at pattern-matching and extraction, they still treat numerical reasoning more as a pattern-matching task than a mathematical one.
Conclusion
The ZXQ team's success at SemEval-2024 demonstrates that fine-tuning general-purpose LLMs like GPT-3.5-Turbo can yield state-of-the-art results in numerical extraction tasks. However, significant hurdles remain regarding complex arithmetic and the identification of "unanswerable" instances where the news content lacks sufficient information. Future work should explore natural language reasoning chains and the inclusion of negative training examples to improve model reliability in real-world news environments.
Summary updated