What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis

Xiang Deng, Vasilisa Bashlovkina, Feng Han, Simon Baumgartner, Michael Bendersky

Introduction

Financial sentiment analysis on social media platforms like Reddit presents a unique challenge for natural language processing. Unlike general sentiment, market sentiment (bullish vs. bearish) requires deep domain knowledge and an understanding of specific jargon such as "diamond hands" or "tendies." This creates a significant "data bottleneck": expert annotation is expensive, agreement between experts is often low (~70%), and the scale of data required for supervised learning is immense.

Researchers from Google and Ohio State University address this by leveraging Large Language Models (LLMs) as expert annotators. By distilling the reasoning of a massive model (PaLM) into a production-friendly student model (Charformer), they demonstrate a path toward high-performance, real-time financial sentiment analysis.

Methodology: From Teacher to Student

The proposed framework utilizes a "Teacher-Student" distillation pipeline. The process begins with an unlabeled social media corpus and follows a multi-step path to create a deployable end model.

Overview of the distillation pipeline and the prompt template used for in-context learning with the LLM teacher.

The Teacher Model (PaLM-540B)

The "Teacher" is a PaLM-540B model used in a few-shot setting. To inject domain knowledge and improve reasoning, the authors employ Chain-of-Thought (CoT) prompting. Specifically, the model is instructed to generate a "TL;DR" summary of the post's financial arguments before predicting the final sentiment. To ensure stability, the researchers use an ensemble approach, running the prompt eight times per post with temperature sampling.

The Student Model (Charformer)

Since massive LLMs are too computationally expensive for real-time monitoring, their knowledge is distilled into a Charformer—a character-level T5 model with approximately 102 million parameters. Instead of training on "hard" binary labels, the student model is trained using a regression loss based on the agreement ratio of the LLM ensemble. This allows the student to learn the inherent uncertainty and nuance captured by the teacher.

Datasets and Evaluation

The study utilizes a new dataset of 20,000 Reddit posts for the distillation process. For evaluation, a "gold-standard" set of 100 posts was annotated by three in-house financial experts. Additionally, the researchers tested the model on the FiQA Benchmark (news headlines and Twitter/Stocktwits microblogs) to assess cross-domain generalization.

Key Insights: The Emergence of Reasoning

A critical finding of the research is the impact of reasoning on model performance. The inclusion of the "TL;DR" summary (CoT) significantly boosted accuracy, particularly for larger models.

Accuracy (ACC) comparison across different model sizes and Chain-of-Thought (COT) iterations.

As shown in the performance analysis, the gain from CoT is much more pronounced in the 540B parameter model compared to the 62B version. This suggests that the ability to reason through complex financial arguments is an emergent property of scale.

Furthermore, training the student model on "soft labels" (agreement ratios) resulted in a smoother precision-recall curve, allowing for better tuning in production environments. Surprisingly, the model trained on Reddit slang generalized well to formal financial news, indicating it captured fundamental financial logic.

Key Results

  • Expert-Level Performance: The PaLM-540B teacher achieved 72% accuracy on Reddit, rivaling the ~70% agreement rate among human experts.
  • Superior Distillation: The 102M parameter Charformer student outperformed specialized models like FinBERT on the Reddit dataset, despite being trained on "weak" synthetic labels.
  • Production Readiness: The final student model maintains the reasoning capabilities of the teacher while being small enough for efficient deployment.

Conclusion and Limitations

While the results are promising, the authors caution against using such models for automated trading due to the risks of market manipulation and the model's occasional struggle with highly contradictory arguments. Future work may involve automatic prompt tuning and the integration of real-time market data to provide better context for the LLM's reasoning.

Summary updated

Abstract

Market sentiment analysis on social media content requires knowledge of both financial markets and social media jargon, which makes it a challenging task for human raters. The resulting lack of high-quality labeled data stands in the way of conventional supervised learning methods. Instead, we approach this problem using semi-supervised learning with a large language model (LLM). Our pipeline generates weak financial sentiment labels for Reddit posts with an LLM and then uses that data to train a small model that can be served in production. We find that prompting the LLM to produce Chain-of-Thought summaries and forcing it through several reasoning paths helps generate more stable and accurate labels, while using a regression loss further improves distillation quality. With only a handful of prompts, the final model performs on par with existing supervised models. Though production applications of our model are limited by ethical considerations, the model's competitive performance points to the great potential of using LLMs for tasks that otherwise require skill-intensive annotation.
BibTeX
@inproceedings{deng2022what,
  author = {Xiang Deng and Vasilisa Bashlovkina and Feng Han and Simon Baumgartner and Michael Bendersky},
  title = {{What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis}},
  booktitle = {The Web Conference},
  year = {2022},
  doi = {10.1145/3543873.3587324},
}
Figure