LLMs for Targeted Sentiment in News Headlines: Exploring Different Levels of Prompt Prescriptiveness

Jana Juros, Laura Majer, Jan Snajder

Abstract

News headlines often evoke sentiment by intentionally portraying entities in particular ways, making targeted sentiment analysis (TSA) of headlines a worthwhile but difficult task. Due to its subjectivity, creating TSA datasets can involve various annotation paradigms, from descriptive to prescriptive, either encouraging or limiting subjectivity. LLMs are a good fit for TSA due to their broad linguistic and world knowledge and in-context learning abilities, yet their performance depends on prompt design. In this paper, we compare the accuracy of state-of-the-art LLMs and fine-tuned encoder models for TSA of news headlines using descriptive and prescriptive datasets across several languages. Exploring the descriptive–prescriptive continuum, we analyze how performance is affected by prompt prescriptiveness, ranging from plain zero-shot to elaborate few-shot prompts. Finally, we evaluate the ability of LLMs to quantify uncertainty via calibration error and comparison to human label variation. We find that LLMs outperform fine-tuned encoders on descriptive datasets, while calibration and F1-score generally improve with increased prescriptiveness, yet the optimal level varies.

Introduction

Targeted Sentiment Analysis (TSA) involves determining the sentiment expressed toward a specific entity within a text. News headlines represent a significant challenge for TSA because they are succinct, often intentionally biased, and highly subjective. This research addresses the Descriptive–Prescriptive Dilemma in data annotation. Descriptive approaches allow for subjective interpretation with minimal instruction, while prescriptive approaches utilize exhaustive rulebooks to ensure consistency. This study explores how LLMs—typically prompted descriptively—perform against datasets built with rigid prescriptive rules and whether increasing prompt prescriptiveness can bridge the performance gap.

Methodology

The study compares state-of-the-art LLMs (including GPT-4, Llama 3, and Phi-3) against fine-tuned encoder models like BERT across three primary datasets: STONE (Croatian, highly prescriptive), SEN (English & Polish, descriptive), and a Spanish dataset (descriptive).

The researchers defined six levels of prompt prescriptiveness:

  • Level 1: Bare-bones task description.
  • Levels 2-4: Increasingly detailed instructions and sentiment definitions.
  • Levels 5-6: Full prescriptive guidelines, including few-shot examples and edge-case explanations.

To measure uncertainty and human-like disagreement, three methods were tested: Self-Consistency Sampling (SCS) (multiple high-temperature prompts), Distribution Prompting (DP) (predicting the distribution of human labels), and Verbal Confidence Assessment (VCA) (numerical confidence scores).

Key Results and Discussion

The Impact of Prompt Prescriptiveness

For high-tier models like GPT-4, performance generally peaked at Level 4. Interestingly, adding specific few-shot examples (Levels 5 and 6) occasionally led to a slight performance dip for GPT models, likely due to sensitivity to example selection. In contrast, smaller open-source models like Llama 3 (8B) showed significant gains at these higher levels, relying more on explicit examples to grasp complex tasks.

F1 Score vs. Calibration (1-ECE) across six levels of prompt prescriptiveness for different uncertainty quantification methods (SCS, DP, VCA).

As shown in Figure 3, both F1 scores and model calibration (the accuracy of internal confidence levels) generally improved as prompts became more prescriptive. The SCS method consistently provided a strong balance between performance and calibration.

LLMs vs. Fine-tuned Encoders

LLMs proved superior on descriptive datasets where subjectivity is expected. However, on the prescriptive Croatian STONE dataset, a fine-tuned BERTic model achieved an F1 of 74.9, outperforming the best LLM (GPT-4 at Level 4) which reached 70.2. This suggests that LLMs possess an "innate" sense of sentiment that is difficult to override solely through prompting.

Modeling Human Disagreement

The study found that the most effective way to mimic human label variation was not asking the model to predict a distribution (DP), but rather using Self-Consistency Sampling (SCS). The stochastic nature of LLM sampling naturally mirrors the subjective "noise" found in human groups.

Comparison of model-predicted majority class votes versus human annotator votes for different models and uncertainty methods.

Conclusion

While LLMs excel at capturing subjective sentiment, matching the precision of prescriptive human annotation requires detailed prompt engineering. Calibration and accuracy generally benefit from increased detail, though the optimal level varies by model size. Future work may explore automated generation of prescriptive guidelines or the use of chain-of-thought reasoning to navigate complex news sentiment edge cases.

Summary updated

Abstract

News headlines often evoke sentiment by intentionally portraying entities in particular ways, making targeted sentiment analysis (TSA) of headlines a worthwhile but difficult task. Due to its subjectivity, creating TSA datasets can involve various annotation paradigms, from descriptive to prescriptive, either encouraging or limiting subjectivity. LLMs are a good fit for TSA due to their broad linguistic and world knowledge and in-context learning abilities, yet their performance depends on prompt design. In this paper, we compare the accuracy of state-of-the-art LLMs and fine-tuned encoder models for TSA of news headlines using descriptive and prescriptive datasets across several languages. Exploring the descriptive--prescriptive continuum, we analyze how performance is affected by prompt prescriptiveness, ranging from plain zero-shot to elaborate few-shot prompts. Finally, we evaluate the ability of LLMs to quantify uncertainty via calibration error and comparison to human label variation. We find that LLMs outperform fine-tuned encoders on descriptive datasets, while calibration and F1-score generally improve with increased prescriptiveness, yet the optimal level varies.
BibTeX
@inproceedings{juros2024llms,
  author = {Jana Juros and Laura Majer and Jan Snajder},
  title = {{LLMs for Targeted Sentiment in News Headlines: Exploring Different Levels of Prompt Prescriptiveness}},
  booktitle = {WASSA},
  year = {2024},
  url = {https://openreview.net/forum?id=h7gT2ZxkNt},
}

Loading related content...

Figure