Team NP_PROBLEM at SemEval-2024 Task 7: Numerical Reasoning in Headline Generation with Preference Optimization

Pawan Rajpoot, Nut ChukamphaengNAACL 2024

TL;DR

This paper tackles the challenge of generating news headlines that accurately incorporate numerical reasoning, a crucial but underexplored aspect of natural language generation. The authors developed a two-stage approach using parameter-efficient fine-tuning of Mistral-7B and Direct Preference Optimization (DPO) to improve numerical accuracy and stylistic alignment, achieving strong results on the SemEval-2024 Task 7.

What Problem Does This Paper Address?

While large language models (LLMs) excel at linguistic tasks, their ability to accurately reason with and integrate numerical information into generated text, particularly in concise formats like news headlines, remains a significant challenge. News headlines often rely on numerical data to convey key information, making accurate numerical reasoning essential for effective summarization and reporting. SemEval-2024 Task 7 specifically addresses this by focusing on numerical headline generation and numerical fill-in-the-blanks, requiring models to not only understand numerical relationships but also to present them in a linguistically fluent and contextually appropriate manner.

Methods

The team approached the SemEval-2024 Task 7 with a two-pronged strategy, developing separate models for each subtask.

For Subtask 1 (Numerical Reasoning - Fill-in-the-Blanks), they employed a novel data augmentation technique. They used GPT-4 to generate rationales explaining how a numerical answer was derived from a given passage. This generated rationale was then used to restructure the training data, creating a richer dataset for fine-tuning their primary model.

For Subtask 2 (Abstractive Headline Generation), they focused on an end-to-end LLM approach. The core of their methodology involved:

  1. Parameter-Efficient Fine-Tuning (PEFT): They selected Mistral-7B, a powerful 7B parameter LLM, and fine-tuned it using techniques like 4-bit QLoRA. This allowed for efficient training while preserving performance.
  2. Direct Preference Optimization (DPO): After initial fine-tuning, they further aligned the models using DPO. This technique directly trains the LLM on human preferences, using pairs of preferred and dispreferred outputs (e.g., a correct headline vs. an incorrect one) to refine the model's ability to generate stylistically appropriate and numerically accurate headlines. For subtask 1, the rationale was also incorporated into the DPO process.

Datasets

The primary dataset used for this work is the NumHG dataset, introduced for SemEval-2024 Task 7. This dataset comprises over 27,000 news articles with detailed annotations specifically designed to facilitate research on numeral-aware headline generation. It includes annotations for various reasoning operators (e.g., Copy, Trans, Span, Round, Add, Subtract, Multiply, Divide) for subtask 1, and news content for subtask 2. The authors also mention using the dev set for alignment during the DPO stage, but this data was not used in the initial training phase.

Key Insights and Ideas

  • Rationale Generation as Data Augmentation: Using a powerful LLM like GPT-4 to generate explanations for numerical reasoning significantly enhances the training data for numerical fill-in-the-blanks tasks. This moves beyond simple input-output pairs to provide deeper contextual understanding.
  • DPO for Stylistic and Numerical Alignment: Direct Preference Optimization proves effective not only for improving numerical accuracy but also for ensuring generated headlines match the desired style and tone of news articles. This is crucial for practical applications.
  • Mistral-7B's Suitability: The choice of Mistral-7B as the base LLM was justified by its strong performance on numerical reasoning benchmarks, indicating its potential for tasks requiring such capabilities.
  • Bridging Numerical Reasoning and Generation: The paper highlights the complex interplay between understanding numerical data and generating coherent, informative text, a gap that SemEval-2024 Task 7 aims to fill.

Key Results

  • Overall Top Numerical Accuracy: The team achieved an overall top numerical accuracy of 73.49% on the SemEval-2024 Task 7.
  • Subtask 1 Performance: For subtask 1, they reported an accuracy of 0.922 for Reasoning and 0.784 for Copy. In comparative SemEval evaluations, they scored 0.89 in the open stage and 0.86 in the hidden stage.
  • Subtask 2 Performance: For subtask 2, they performed best in numerical accuracy overall and reasoning scores, with a difference of approximately 7 points between their rank and the second-ranked team in reasoning accuracy.
  • Fine-tuning Efficiency: Mistral-7B demonstrated lower fine-tuning loss compared to BART, reinforcing its suitability for the task.

Limitations and Future Work

  • LLM Size Constraint: The performance of their method is inherently limited by the capabilities of the 7B parameter LLM used. Larger models might offer further improvements.
  • Hardware Resources: The fine-tuning process was constrained by available hardware, specifically an RTX 4090 GPU, which limits the size of models that can be effectively fine-tuned.
  • Future Directions: The authors suggest that further research could explore more advanced alignment modules to enhance the model's ability to understand numerical reasoning. They also open-source their final models on Huggingface, encouraging further development in this area.

Summary updated

BibTeX
@inproceedings{rajpoot2024team,
  author = {Pawan Rajpoot and Nut Chukamphaeng},
  title = {{Team NP\_PROBLEM at SemEval-2024 Task 7: Numerical Reasoning in Headline Generation with Preference Optimization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure