ClusterCore at SemEval-2024 Task 7: Few Shot Prompting With Large Language Models for Numeral-Aware Headline Generation

Monika Singh, Sujit Kumar, Tanveen, Sanasam Ranbir SinghNAACL 2024

TL;DR

This paper tackles the challenge of generating accurate news headlines that include numerical information, a task where standard models often falter. The authors propose and evaluate a few-shot prompting approach using Large Language Models (LLMs) to improve numeral-aware headline generation, demonstrating that this method outperforms fine-tuning LLMs on this specific task.

What Problem Does This Paper Address?

News headlines are crucial for summarizing articles and driving engagement. However, generating headlines that accurately incorporate numerical data (e.g., percentages, counts, dates) is a persistent challenge for automated systems. While modern encoder-decoder and transformer-based models have shown promise in abstractive summarization, they often struggle with the precise representation of numerical figures within headlines. This can lead to misleading or inaccurate summaries, impacting the credibility of news. The paper addresses this gap by focusing on "numeral-aware headline generation," aiming to ensure that generated headlines not only capture the essence of an article but also correctly convey its numerical information.

Methods

The core of this work lies in exploring and proposing a few-shot prompting strategy for Large Language Models (LLMs) to achieve numeral-aware headline generation. The authors investigate three key components:

  • Large Language Models (LLMs): They utilize prominent LLMs like GPT, T5, and LLaMA, evaluating their performance in both pre-trained and fine-tuned states.
  • Base Prompt: A general instruction is provided to the LLMs to guide them in generating headlines. This prompt specifies that the headline should be concise, represent the article's content, and may include numbers derived from arithmetic operations or directly from the text.
  • Few-Shot Prompting: This is the central contribution. Instead of solely relying on fine-tuning, the authors employ in-context learning, where a limited number of examples (shots) are provided to the LLM at inference time. This guides the model's output by demonstrating the desired format and content. They explore zero-shot (no examples), one-shot (one example), and few-shot (multiple examples) prompting strategies.

Datasets

The study utilizes two primary datasets:

  • NumHG dataset: Curated by Huang et al. (2023), this dataset is used for training the models.
  • NumEval-2024 Task-3(2) test set: Provided by the organizers of NumEval-2024, this dataset is used for evaluating the performance of the proposed methods.

The paper also mentions a subset of fifty news headline-body pairs, specifically selected for manual inspection where numeral figures in the headline are critical for accurate representation.

Key Insights and Ideas

  • LLMs are powerful, but need guidance for numerical accuracy: While LLMs show strong general text generation capabilities, they don't inherently excel at precise numerical representation in headlines without specific prompting.
  • Few-shot prompting is more effective than fine-tuning for this task: The study's key finding is that few-shot prompting with LLMs significantly outperforms fine-tuned LLMs for numeral-aware headline generation. This suggests that providing examples at inference time is a more direct way to steer LLMs towards this specific, nuanced task.
  • LLaMA shines with few-shot prompting: Among the LLMs tested, LLaMA demonstrated particularly strong performance when combined with few-shot prompting, outperforming GPT and T5 in this setup.
  • Error analysis reveals model weaknesses: The error analysis highlighted that some models (like RoBERTa) often fail to consider numerical values, while others (BART, T5) are better but still not perfect. LLaMA, especially with few-shot prompting, showed the most promise in generating accurate numerical headlines.

Key Results

  • Fine-tuning improves performance: Fine-tuning LLMs on the NumHG dataset generally enhances their performance in numeral-aware headline generation compared to their pre-trained state.
  • Few-shot prompting surpasses fine-tuning: The paper's main quantitative result is that few-shot prompting with LLMs achieved better performance than fine-tuned LLMs for numeral-aware headline generation.
  • LLaMA with few-shot prompting achieved top rank: The proposed system, utilizing few-shot prompting with LLaMA, achieved the top rank in the NumEval-2024 Task 7 evaluation for recommending news headlines.
  • Qualitative improvements: Figure 2 illustrates that a T5 model with three-shot prompting generated a headline that better represented the news body compared to a fine-tuned T5 model, visually demonstrating the benefit of few-shot prompting.

Limitations and Future Work

  • Resource Constraints: The authors mention that they could have fine-tuned the LLaMA model for potentially better results but were limited by resource constraints, opting to use the pre-trained LLaMA model with few-shot prompting.
  • Future Work: The paper identifies prompt tuning using LLMs as a promising direction for numeral-aware headline generation. Further research could explore more sophisticated prompt engineering techniques and investigate the optimal number of shots for different LLMs and tasks.

Summary updated

BibTeX
@inproceedings{singh2024clustercore,
  author = {Monika Singh and Sujit Kumar and Tanveen and Sanasam Ranbir Singh},
  title = {{ClusterCore at SemEval-2024 Task 7: Few Shot Prompting With Large Language Models for Numeral-Aware Headline Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure