The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges

Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, Jimin HuangarXiv 2023 · DOI · arXiv

Abstract

While Large Language Models (LLMs) have achieved remarkable success across various NLP benchmarks, their efficacy in specialized, high-stakes domains like financial forecasting remains an open question. This paper investigates the performance of ChatGPT in a zero-shot setting for multimodal stock movement prediction, comparing it against traditional machine learning and state-of-the-art specialized models.

Introduction

Predicting stock market volatility is a "holy grail" task that traditionally relies on two distinct data streams: numerical historical prices and textual information from news or social media. While specialized deep learning models have been developed for this multimodal task, the rise of general-purpose LLMs like ChatGPT prompts a critical inquiry: Can a model trained on a massive, general corpus act as an effective financial analyst without domain-specific fine-tuning? This study explores ChatGPT’s ability to fuse tabular price data with social media sentiment (Twitter/X) to predict stock price direction.

Methodology

The researchers evaluated ChatGPT (gpt-3.5-turbo) using two primary zero-shot prompting strategies:

  1. Vanilla Zero-Shot: The model is provided with historical price features and concatenated tweets, then asked for a binary prediction ("Rise" or "Fall").
  2. Chain-of-Thought (CoT): The model is prompted to "explain your predictions step by step" before arriving at a final answer, leveraging its inherent reasoning capabilities.

Performance was benchmarked against a variety of models, including basic Logistic Regression (LR), Random Forests (RF), specialized architectures like LSTM and Adv-ALSTM, and the current state-of-the-art (SOTA) multimodal model, SLOT.

Datasets

The study utilized three established benchmark datasets:

  • BIGDATA22: 50 stocks, ~272,000 tweets (2019–2020).
  • ACL18: 87 stocks, ~106,000 tweets (2014–2015).
  • CIKM18: 38 stocks, ~955,000 tweets (2017).

Experimental Results

The experimental results indicate that ChatGPT currently functions as a "Wall Street Neophyte." It consistently underperformed specialized models like SLOT and even struggled against simpler linear models in certain metrics.

Table 2: Performance comparison of ChatGPT against baseline models across three datasets.

Key findings from the results include:

  • Accuracy and Correlation: On the BIGDATA22 dataset, SLOT achieved 54.81% accuracy, while ChatGPT reached 53.13%. More tellingly, ChatGPT’s Matthews Correlation Coefficient (MCC) was -0.0251, suggesting its predictions were near-random, whereas SLOT maintained a positive 0.0952.
  • The Value of Tweets: Incorporating social media sentiment consistently improved ChatGPT’s accuracy. Removing tweets from the prompt led to a 2.45% drop in accuracy on BIGDATA22, indicating that ChatGPT is a capable sentiment analyst even if it lacks financial depth.
  • The Prompting Paradox: Interestingly, Chain-of-Thought (CoT) prompting did not consistently improve performance. In some instances, the "explanation" phase introduced errors or led the model to over-justify incorrect predictions.

Qualitative Analysis: Superficial Reasoning

Case studies revealed that ChatGPT’s financial reasoning is often shallow. It tends to fixate on simple linear trends and struggles to identify complex trend reversals or weigh conflicting signals between price data and sentiment.

Table 3: Example of ChatGPT's prompt and response, illustrating its reasoning process and final prediction.

The model often treats numerical and textual data as a single sequence rather than performing sophisticated cross-modal fusion. This lack of specialized attention limits its ability to correlate specific social media signals with precise price movements.

Conclusion

While ChatGPT demonstrates a baseline ability to process financial data and sentiment, it remains a "neophyte" compared to specialized multimodal models. Future work should focus on specialized fine-tuning (e.g., on financial corpora like BloombergGPT) and developing hybrid architectures that combine the linguistic power of LLMs with the quantitative precision of traditional financial models.

Summary updated

Abstract

Recently, large language models (LLMs) like ChatGPT have demonstrated remarkable performance across a variety of natural language processing tasks. However, their effectiveness in the financial domain, specifically in predicting stock market movements, remains to be explored. In this paper, we conduct an extensive zero-shot analysis of ChatGPT's capabilities in multimodal stock movement prediction, on three tweets and historical stock price datasets. Our findings indicate that ChatGPT is a "Wall Street Neophyte" with limited success in predicting stock movements, as it underperforms not only state-of-the-art methods but also traditional methods like linear regression using price features. Despite the potential of Chain-of-Thought prompting strategies and the inclusion of tweets, ChatGPT's performance remains subpar. Furthermore, we observe limitations in its explainability and stability, suggesting the need for more specialized training or fine-tuning. This research provides insights into ChatGPT's capabilities and serves as a foundation for future work aimed at improving financial market analysis and prediction by leveraging social media sentiment and historical stock data.
BibTeX
@misc{xie2023wall,
  author = {Qianqian Xie and Weiguang Han and Yanzhao Lai and Min Peng and Jimin Huang},
  title = {{The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges}},
  year = {2023},
  eprint = {2304.05351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.05351},
}
Figure