NumDecoders at SemEval-2024 Task 7: FlanT5 and GPT enhanced with CoT for Numerical Reasoning

Andres Gonzalez, Md Zobaer Hossain, Jahedul Alam JunaedNAACL 2024

TL;DR

This paper tackles numerical reasoning in news headlines by enhancing large language models (LLMs) like FlanT5 and GPT-3.5 with Chain-of-Thought (CoT) prompting and a novel data augmentation strategy. They achieved a 91% accuracy on the NumEval Task 7, demonstrating the effectiveness of explicit reasoning steps, though larger models showed more significant gains.

What Problem Does This Paper Address?

The paper addresses the challenge of quantitative reasoning for Large Language Models (LLMs), specifically in the context of filling in numerical blanks in news headlines. This task, known as NumEval, requires models to perform mathematical calculations based on information presented in a passage to derive a numerical answer. Existing methods often struggle with the nuances of numerical reasoning, especially when multiple operations or implicit contextual clues are involved. The goal is to improve LLM performance on these tasks by providing them with more structured reasoning capabilities.

Methods

The core of the approach involves enhancing LLMs with Chain-of-Thought (CoT) prompting and a data augmentation strategy.

  • Chain-of-Thought (CoT) Prompting: This technique encourages models to generate intermediate reasoning steps before arriving at the final answer. The authors implemented this by explicitly adding these reasoning steps to the training data.
  • Data Augmentation: They augmented the original NumEval dataset with additional mathematical reasoning observations. This involved:
  • Automatic Annotation: Developing a system to automatically extract and normalize dates from news articles and transform questions from the DROP dataset into a fill-in-the-blank format.
  • Manual Annotation: Employing human annotators to refine and correct the automatically generated reasoning steps, especially for more complex calculations, ensuring consistency and accuracy.
  • Model Architectures: They experimented with FlanT5 (in small, base, and large versions) and GPT-3.5 Turbo.
  • Prompt Engineering: For GPT-3.5, they utilized specific prompt patterns (Persona, Context Manager, Recipe) to guide the model's output and focus its reasoning.
  • Ensemble Majority Voting: To achieve the final prediction, they employed an ensemble of their best-performing models, selecting the answer with the most votes.

Datasets

  • NumHG Dataset: The primary dataset for numerical headline generation, used for training and evaluation.
  • DROP Dataset (Discrete Reasoning Over the Content of Paragraph): This dataset was merged with NumHG to augment the training data with additional numerical reasoning examples. The authors filtered it to include only numerical reasoning tasks and transformed its question-answer format to match the headline generation task.

Key Insights and Ideas

  • CoT is Crucial for Numerical Reasoning: Explicitly providing intermediate reasoning steps significantly aids LLMs in solving numerical problems, aligning with previous findings.
  • Model Size Matters for CoT Gains: Larger LLMs benefit more from CoT prompting and data augmentation than smaller ones. Smaller models, in this study, did not show significant improvements and sometimes even saw a decrease in accuracy.
  • Data Quality is Paramount: Inconsistent or erroneous annotations in the original dataset posed a significant challenge. Manual annotation and data cleanup were essential for improving model performance.
  • Ensembling Boosts Performance: Combining the predictions of multiple well-trained models through majority voting led to the highest accuracy.

Key Results

  • The team achieved 91% accuracy on the NumEval Task 7, ranking 5th out of 16 teams.
  • Their majority voting ensemble model achieved 91% accuracy on the test set.
  • The best-performing FlanT5 models were the large versions, especially when trained on the NumHG dataset.
  • A fine-tuned GPT-3.5 Turbo model underperformed compared to some FlanT5 models, despite its larger size.
  • Error analysis revealed that complex operations like addition, subtraction, and rounding had higher error rates.

Limitations and Future Work

  • Model Size Limitation: The study primarily used smaller and base versions of FlanT5, which did not exhibit the significant gains seen in larger models (100B+ parameters) in previous CoT research. The authors suggest that experimenting with XL and XXL versions of these models could yield much greater improvements.
  • Annotation Inconsistencies: The paper acknowledges that automatic annotations sometimes have issues compared to manual ones, and the original dataset contained ambiguous or erroneous entries.
  • Future Work:
    • Crowdsourcing for Annotations: Improving automatic annotations through crowdsourcing methods is suggested to achieve more accurate results.
  • Data Cleanup: Emphasizing the necessity of data cleanup to enhance mathematical reasoning in language models.
  • Leveraging Larger LLMs: The authors argue that larger LLMs could lead to even greater gains in quantitative reasoning tasks when combined with CoT fine-tuning.

Summary updated

BibTeX
@inproceedings{gonzalez2024numdecoders,
  author = {Andres Gonzalez and Md Zobaer Hossain and Jahedul Alam Junaed},
  title = {{NumDecoders at SemEval-2024 Task 7: FlanT5 and GPT enhanced with CoT for Numerical Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure