Infrrd.ai at SemEval-2024 Task 7: RAG-based end-to-end training to generate headlines and numbers

Jianglong He, Saiteja Tallam, Srirama Nakshathri, Navaneeth Amarnath, Pratiba Kr, Deepak KumarNAACL 2024

TL;DR

This paper introduces a novel Retrieval-Augmented Generation (RAG) approach for end-to-end training of Large Language Models (LLMs) to generate news headlines that not only summarize content but also accurately incorporate and reason about numerical data. By leveraging RAG to retrieve relevant examples and employing Chain-of-Thought (CoT) prompting for numerical operations, the model achieves improved headline generation and numerical accuracy, outperforming baseline methods.

What Problem Does This Paper Address?

In today's information-saturated world, efficiently consuming news is a challenge. Headlines play a crucial role in attracting readers, and often, they condense complex information into a few words, frequently including numbers to highlight key facts. However, generating accurate and informative headlines that correctly handle numerical data, especially when mathematical reasoning is involved, remains a difficult task for LLMs. Existing benchmarks often focus on either general summarization or numerical reasoning in isolation, leaving a gap in evaluating models that can perform both tasks concurrently and accurately within the context of headline generation. This paper tackles the problem of generating headlines that are not only semantically relevant but also numerically sound, requiring models to understand, extract, and potentially reason with numbers present in the source text.

Methods

The core of the proposed method is a Retrieval-Augmented Generation (RAG) framework trained end-to-end. This system comprises three main modules:

  • Knowledge Base (KB): A vector database is constructed using pre-trained BERT embeddings of news articles from the training dataset. This KB allows for efficient similarity search.
  • Dense Retrieval: When presented with an input news article, the system encodes it into a vector and uses this to retrieve the top-k most similar news articles from the KB. These retrieved examples serve as context for the generative model.
  • Generation Module: The LLAMA2-7b model is employed for headline generation. The retrieved examples are integrated into a specific template along with the input news to guide the LLM. This "in-context learning" approach aims to provide the model with similar examples to improve its output.

To enhance numerical reasoning, the authors incorporate two key techniques:

  • Chain-of-Thought (CoT) Prompting: For numerical operations, the model is prompted to generate a series of reasoning steps. This is achieved by augmenting the ground truth headlines with XML tags (<NR>) that encapsulate numerical reasoning, often in the form of mathematical operations (e.g., paraphrase(Add(100000,359000),K)). This encourages the model to perform calculations explicitly.
  • Verification Step: A post-processing step is introduced to verify the numerical accuracy of the generated headlines. This involves checking if the numbers and calculations within the XML tags are correct and present in the original text. If verification fails, a fallback mechanism using a BRIO model is employed to replace the headline.

Datasets

The primary dataset used in this work is NumHG, which provides news articles and corresponding headlines, including numerical reasoning tasks. The authors also mention the NumEval dataset, which is used for evaluating numerical reasoning capabilities.

Key Insights and Ideas

  • RAG for Contextual Numerical Reasoning: The paper demonstrates that RAG is effective not only for providing relevant examples for headline generation but also for guiding LLMs in performing numerical reasoning within that context.
  • XML Tags as Numerical Anchors: Using XML tags to explicitly mark numerical information and associated reasoning tasks significantly helps the model focus on and process these elements.
  • CoT for Explicit Calculation: Chain-of-Thought prompting proves crucial for enabling LLMs to perform mathematical operations required for accurate numerical inclusion in headlines.
  • Verification is Key for Trustworthiness: A post-processing verification step is vital for ensuring the factual accuracy of numerical data in generated headlines, especially when LLMs might hallucinate or miscalculate.
  • LLAMA2-7b Shows Promise: The LLAMA2-7b model, when fine-tuned with the proposed RAG and CoT approach, shows competitive performance, particularly in numerical reasoning.

Key Results

The paper reports results across several tables, highlighting the effectiveness of their proposed methods.

  • Headline Generation (ROUGE Scores): While not explicitly stated as the best in all tables, the RAG-based approaches generally show improvement over simpler prompts. For instance, in Table 3 (in-context learning), models using RAG achieve ROUGE-L scores in the mid-30s, outperforming the out-of-the-box LLAMA2-7b. Table 4 shows that the RAG-based end-to-end approach with XML tags and post-processing achieves a ROUGE-L score of 43.32, indicating strong performance.
  • Numerical Reasoning Accuracy: Table 5 shows that fine-tuning LLAMA2-7b significantly boosts numerical reasoning accuracy from 6.53% (out-of-the-box) to 86.31%. The Chain-of-Thought (COT) approach further improves this to 89.54%, and a combined COT with minority sampling reaches 91.44%.
  • Human Evaluation: In human assessment (Table 8), the proposed method ranks highly, achieving the top score for "Recommendation" (100 News) with a score of 1.73 and a strong second place for "Num Acc." (50 Headlines) with a score of 1.73.

Limitations and Future Work

  • Generalization of Mathematical Operations: The authors note that while CoT improves accuracy, the model may not always complete all steps of a calculation, and the distribution of mathematical operations in datasets can be narrow. They suggest exploring further the rationale steps followed by the model in CoT and improving performance by taking advantage of mathematical operations.
  • Ensemble Approaches: The paper mentions attempting to combine results from multiple models but acknowledges that a full exploration of ensemble approaches was not undertaken.
  • Overfitting Concerns: Fine-tuning the model to confirm ground truth might lead to overfitting, potentially deviating from generalization capabilities. The authors speculate that generating multiple headlines and using human feedback could offer better insights into the model's robustness.
  • Computational Cost: While not explicitly detailed as a limitation, the use of LLMs and RAG implies significant computational resources are required for training and inference.

Future work could involve exploring more complex mathematical operations, investigating ensemble methods for improved robustness, and developing more sophisticated methods for handling numerical reasoning in diverse contexts.

Summary updated

BibTeX
@inproceedings{he2024infrrd,
  author = {Jianglong He and Saiteja Tallam and Srirama Nakshathri and Navaneeth Amarnath and Pratiba Kr and Deepak Kumar},
  title = {{Infrrd.ai at SemEval-2024 Task 7: RAG-based end-to-end training to generate headlines and numbers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure