Introduction
Financial sentiment analysis is more complex than general sentiment tasks; it requires determining whether a post is bullish (expecting prices to rise) or bearish (expecting them to fall). A major challenge in social media finance (e.g., Stocktwits) is the discrepancy between a writer’s intent and a reader’s perception. This requires Theory of Mind (ToM)—the ability to attribute mental states to others to understand sarcasm or technical jargon. While Large Language Models (LLMs) show general ToM capabilities, they often struggle to apply them in specialized domains like finance, where numerical data is the primary language of the expert.
Methodology
The researchers compared three prompting strategies across four state-of-the-art LLMs (PaLM 2, Gemini Pro, GPT-3.5, and GPT-4):
- Simple Prompt: Direct classification.
- Chain-of-Thought (CoT): Providing an analytical reasoning step before the final prediction.
- CoT + Hint: Adding a specific "expert hint": "Focus particularly on any numerical data or statistics present in the tweet, as these figures may be crucial in determining the overall sentiment."
Datasets
The study utilized the Fin-SoMe dataset (~10,000 tweets), featuring labels from both writers and readers. A "Perspective-Taking" subset (1,834 tweets) was isolated, containing cases where the writer intended a sentiment but readers perceived it as neutral. These represent the hardest cases for sentiment analysis. Additionally, the FinNum dataset was used to categorize numerical types (e.g., Monetary, Temporal) to identify which data points benefited most from the hint.
Key Results and Insights
The "CoT + Hint" method consistently outperformed standard CoT, suggesting that LLMs lack the intrinsic intuition to prioritize numerical cues without explicit instruction.
- Universal Improvement: The expert hint improved Micro-F1 and Weighted-F1 scores across nearly all models and subsets.
- Significant Gains in Perspective-Taking: The most dramatic improvements occurred in the hardest subset. For example, GPT-3.5’s Micro-F1 jumped from 43.35 (CoT) to 65.27 (CoT + Hint).
- The Power of Monetary Data: Analysis by number category revealed that "Monetary" figures saw the highest performance gains (up to 59.37% for GPT-3.5), confirming that "following the money" is the most effective heuristic for financial sentiment.
Conclusion
LLMs are not "natural" financial experts, but they can be nudged toward expert-level performance. By directing models to focus on numerical data, researchers successfully activated the models' perspective-taking abilities. Future work may explore other expert-designed hints, such as linguistic subtleties or cultural references, to further refine LLM performance in specialized domains.
Summary updated