LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma

Jana Juros, Laura Majer, Jan Snajder

TL;DR

This paper investigates how Large Language Models (LLMs) perform on Targeted Sentiment Analysis (TSA) of news headlines, comparing them to traditional fine-tuned models. They found that LLMs excel with more descriptive datasets and that increasing prompt prescriptiveness generally improves LLM performance, though the optimal level varies. This work is crucial for understanding how to best leverage LLMs for nuanced sentiment analysis tasks.

What Problem Does This Paper Address?

News headlines are designed to be attention-grabbing and often use specific language to evoke sentiment towards entities mentioned within them. This makes Targeted Sentiment Analysis (TSA) – determining the sentiment expressed towards a specific entity – a challenging but important task. A key difficulty in TSA is the inherent subjectivity involved. Researchers have explored different ways to create datasets for TSA, broadly categorized as "descriptive" (encouraging diverse interpretations with minimal guidelines) and "prescriptive" (providing detailed instructions to limit subjectivity).

LLMs, with their vast knowledge, are promising for TSA, but their performance heavily relies on how they are prompted. This paper aims to bridge the gap between annotation paradigms and LLM prompting by:

  1. Comparing LLMs and fine-tuned encoder models (like BERT) on TSA datasets annotated with both descriptive and prescriptive approaches.
  2. Investigating how the level of detail and specificity in LLM prompts (from zero-shot to few-shot) affects their accuracy.
  3. Evaluating LLMs' ability to quantify their own uncertainty and how this relates to human annotator disagreement.

Methods

The researchers compared the performance of several LLMs (GPT-4, GPT-3.5 Turbo, Neural Chat, Llama 3, Phi-3, Gemma) against fine-tuned BERT models. They employed a zero-shot and few-shot prompting strategy for the LLMs.

A core part of their methodology involved exploring the "descriptive-prescriptive continuum" of prompts. They created six levels of prompt prescriptiveness, starting from a very basic instruction and progressively adding more detail, mirroring the guidelines used in human annotation. This allowed them to systematically study how prompt specificity impacts LLM performance.

To assess LLM uncertainty, they used three methods:

  • Self-consistency sampling (SCS): Repeatedly prompting the model for the same input to capture its inherent stochasticity.
  • Distribution prompting (DP): Asking the model to explicitly predict how multiple annotators would label the sentiment.
  • Verbal confidence assessment (VCA): Prompting the model to provide confidence scores for each sentiment class.

They then evaluated model calibration (how well predicted confidence aligns with actual accuracy) using Expected Calibration Error (ECE) and compared LLM label distributions to human label variation.

Datasets

The study utilized three datasets for TSA on news headlines:

  • STONE: A Croatian dataset annotated with detailed, prescriptive guidelines.
  • SEN: A dataset covering English and Polish headlines, annotated with vaguer, descriptive guidelines.
  • Spanish (ES): A dataset of headlines related to the 2019 Argentinian Presidential Elections, with undisclosed annotation guidelines but likely leaning towards descriptive.

Key Insights and Ideas

  • LLMs vs. Fine-tuned Models: LLMs generally outperformed fine-tuned BERT models on the more descriptive datasets (SEN, ES), suggesting they are adept at capturing the nuances of less constrained annotation. However, on the prescriptive STONE dataset, BERTić (a Croatian BERT model) achieved the highest F1 score, indicating that fine-tuned models can still be superior when highly specific guidelines are provided.
  • The Power of Prescriptive Prompts: For LLMs, increasing the prescriptiveness of prompts (i.e., providing more detailed instructions) generally led to improved predictive accuracy and calibration. This highlights the importance of prompt engineering for achieving optimal LLM performance, mirroring the need for clear guidelines in human annotation.
  • Optimal Prescriptiveness Varies: While more prescriptive prompts were generally better, the "optimal" level of prescriptiveness was not uniform across all models or uncertainty quantification methods. This suggests a need for tailored prompt design based on the specific LLM and task.
  • Uncertainty and Human Variation: Self-consistency sampling (SCS) showed the strongest alignment with human label variation, suggesting it's a promising method for modeling the inherent subjectivity in TSA.
  • Calibration Matters: GPT-4 demonstrated strong performance not only in accuracy but also in calibration across different prompt levels and uncertainty methods, indicating a more reliable and trustworthy output.

Key Results

  • F1 Scores: LLMs generally achieved higher F1 scores on descriptive datasets (SEN, ES) compared to BERT. BERTić outperformed all LLMs on the prescriptive STONE dataset.
  • Prompt Prescriptiveness: Performance generally improved from prompt levels 1 to 4, with a slight drop at levels 5 and 6 (few-shot prompts), possibly due to sensitivity to example selection.
  • Uncertainty Quantification: SCS generally yielded higher F1 scores compared to DP and VCA when evaluated against gold labels. Calibration accuracy was generally high across methods, with optimal prescriptiveness levels varying.
  • GPT-4 Dominance: GPT-4 consistently showed the best performance, with high F1 scores and stable calibration across various settings.

Limitations and Future Work

  • Limited LLM Selection: The study's LLM selection was constrained by computational resources. A broader range of models, especially larger open-source ones, would provide a more comprehensive overview.
  • Prompting Costs: Prompting GPT models with batches of data was found to be computationally expensive, impacting the ability to test every instance individually.
  • Single Prescriptive Dataset: The exploration of prompt prescriptiveness was primarily conducted on the STONE dataset. The lack of other publicly available, richly annotated prescriptive TSA datasets limits generalizability.
  • Language Specificity: The STONE dataset being in Croatian raises questions about whether performance differences are due to language proficiency or the inherent nature of the task and annotation.
  • Arbitrary Prompt Levels: The six levels of prompt prescriptiveness were chosen somewhat arbitrarily, and different segmentation or generation methods could yield different results.

Future work could involve expanding the LLM selection, exploring more diverse prescriptive datasets, and investigating the impact of prompt design across a wider range of languages and domains. The authors also suggest further research into how LLMs can better model human label variation.

Summary updated

Abstract

News headlines often evoke sentiment by intentionally portraying entities in particular ways, making targeted sentiment analysis (TSA) of headlines a worthwhile but difficult task. Due to its subjectivity, creating TSA datasets can involve various annotation paradigms, from descriptive to prescriptive, either encouraging or limiting subjectivity. LLMs are a good fit for TSA due to their broad linguistic and world knowledge and in-context learning abilities, yet their performance depends on prompt design. In this paper, we compare the accuracy of state-of-the-art LLMs and fine-tuned encoder models for TSA of news headlines using descriptive and prescriptive datasets across several languages. Exploring the descriptive–prescriptive continuum, we analyze how performance is affected by prompt prescriptiveness, ranging from plain zero-shot to elaborate few-shot prompts. Finally, we evaluate the ability of LLMs to quantify uncertainty via calibration error and comparison to human label variation. We find that LLMs outperform fine-tuned encoders on descriptive datasets, while calibration and F1-score generally improve with increased prescriptiveness, yet the optimal level varies.
BibTeX
@inproceedings{juros2024llms,
  author = {Jana Juros and Laura Majer and Jan Snajder},
  title = {{LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemma}},
  booktitle = {WASSA},
  year = {2024},
  url = {https://openreview.net/forum?id=h7gT2ZxkNt},
}
Figure