Enhancing Financial Sentiment Analysis with Expert-Designed Hint

Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke MiyaoarXiv 2024 · DOI · arXiv

Introduction

Financial sentiment analysis is more complex than general sentiment tasks; it requires determining whether a post is bullish (expecting prices to rise) or bearish (expecting them to fall). A major challenge in social media finance (e.g., Stocktwits) is the discrepancy between a writer’s intent and a reader’s perception. This requires Theory of Mind (ToM)—the ability to attribute mental states to others to understand sarcasm or technical jargon. While Large Language Models (LLMs) show general ToM capabilities, they often struggle to apply them in specialized domains like finance, where numerical data is the primary language of the expert.

Methodology

The researchers compared three prompting strategies across four state-of-the-art LLMs (PaLM 2, Gemini Pro, GPT-3.5, and GPT-4):

  1. Simple Prompt: Direct classification.
  2. Chain-of-Thought (CoT): Providing an analytical reasoning step before the final prediction.
  3. CoT + Hint: Adding a specific "expert hint": "Focus particularly on any numerical data or statistics present in the tweet, as these figures may be crucial in determining the overall sentiment."
A case study showing how the expert hint shifts model reasoning from neutral to bullish by focusing on purchase prices.

Datasets

The study utilized the Fin-SoMe dataset (~10,000 tweets), featuring labels from both writers and readers. A "Perspective-Taking" subset (1,834 tweets) was isolated, containing cases where the writer intended a sentiment but readers perceived it as neutral. These represent the hardest cases for sentiment analysis. Additionally, the FinNum dataset was used to categorize numerical types (e.g., Monetary, Temporal) to identify which data points benefited most from the hint.

Key Results and Insights

The "CoT + Hint" method consistently outperformed standard CoT, suggesting that LLMs lack the intrinsic intuition to prioritize numerical cues without explicit instruction.

Performance metrics across models show that the CoT + Hint approach yields the highest F1 scores, particularly in the Perspective-Taking subset.
  • Universal Improvement: The expert hint improved Micro-F1 and Weighted-F1 scores across nearly all models and subsets.
  • Significant Gains in Perspective-Taking: The most dramatic improvements occurred in the hardest subset. For example, GPT-3.5’s Micro-F1 jumped from 43.35 (CoT) to 65.27 (CoT + Hint).
  • The Power of Monetary Data: Analysis by number category revealed that "Monetary" figures saw the highest performance gains (up to 59.37% for GPT-3.5), confirming that "following the money" is the most effective heuristic for financial sentiment.

Conclusion

LLMs are not "natural" financial experts, but they can be nudged toward expert-level performance. By directing models to focus on numerical data, researchers successfully activated the models' perspective-taking abilities. Future work may explore other expert-designed hints, such as linguistic subtleties or cultural references, to further refine LLM performance in specialized domains.

Summary updated

Abstract

This paper investigates the role of expert-designed hint in enhancing sentiment analysis on financial social media posts. We explore the capability of large language models (LLMs) to empathize with writer perspectives and analyze sentiments. Our findings reveal that expert-designed hint, i.e., pointing out the importance of numbers, significantly improve performances across various LLMs, particularly in cases requiring perspective-taking skills. Further analysis on tweets containing different types of numerical data demonstrates that the inclusion of expert-designed hint leads to notable improvements in sentiment analysis performance, especially for tweets with monetary-related numbers. Our findings contribute to the ongoing discussion on the applicability of Theory of Mind in NLP and open new avenues for improving sentiment analysis in financial domains through the strategic use of expert knowledge.
BibTeX
@misc{chen2024enhancing,
  author = {Chung-Chi Chen and Hiroya Takamura and Ichiro Kobayashi and Yusuke Miyao},
  title = {{Enhancing Financial Sentiment Analysis with Expert-Designed Hint}},
  year = {2024},
  eprint = {2409.17448},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.17448},
}
Figure