Abstract
While Large Language Models (LLMs) have achieved remarkable success across various NLP benchmarks, their efficacy in specialized, high-stakes domains like financial forecasting remains an open question. This paper investigates the performance of ChatGPT in a zero-shot setting for multimodal stock movement prediction, comparing it against traditional machine learning and state-of-the-art specialized models.
Introduction
Predicting stock market volatility is a "holy grail" task that traditionally relies on two distinct data streams: numerical historical prices and textual information from news or social media. While specialized deep learning models have been developed for this multimodal task, the rise of general-purpose LLMs like ChatGPT prompts a critical inquiry: Can a model trained on a massive, general corpus act as an effective financial analyst without domain-specific fine-tuning? This study explores ChatGPT’s ability to fuse tabular price data with social media sentiment (Twitter/X) to predict stock price direction.
Methodology
The researchers evaluated ChatGPT (gpt-3.5-turbo) using two primary zero-shot prompting strategies:
- Vanilla Zero-Shot: The model is provided with historical price features and concatenated tweets, then asked for a binary prediction ("Rise" or "Fall").
- Chain-of-Thought (CoT): The model is prompted to "explain your predictions step by step" before arriving at a final answer, leveraging its inherent reasoning capabilities.
Performance was benchmarked against a variety of models, including basic Logistic Regression (LR), Random Forests (RF), specialized architectures like LSTM and Adv-ALSTM, and the current state-of-the-art (SOTA) multimodal model, SLOT.
Datasets
The study utilized three established benchmark datasets:
- BIGDATA22: 50 stocks, ~272,000 tweets (2019–2020).
- ACL18: 87 stocks, ~106,000 tweets (2014–2015).
- CIKM18: 38 stocks, ~955,000 tweets (2017).
Experimental Results
The experimental results indicate that ChatGPT currently functions as a "Wall Street Neophyte." It consistently underperformed specialized models like SLOT and even struggled against simpler linear models in certain metrics.
Key findings from the results include:
- Accuracy and Correlation: On the BIGDATA22 dataset, SLOT achieved 54.81% accuracy, while ChatGPT reached 53.13%. More tellingly, ChatGPT’s Matthews Correlation Coefficient (MCC) was -0.0251, suggesting its predictions were near-random, whereas SLOT maintained a positive 0.0952.
- The Value of Tweets: Incorporating social media sentiment consistently improved ChatGPT’s accuracy. Removing tweets from the prompt led to a 2.45% drop in accuracy on BIGDATA22, indicating that ChatGPT is a capable sentiment analyst even if it lacks financial depth.
- The Prompting Paradox: Interestingly, Chain-of-Thought (CoT) prompting did not consistently improve performance. In some instances, the "explanation" phase introduced errors or led the model to over-justify incorrect predictions.
Qualitative Analysis: Superficial Reasoning
Case studies revealed that ChatGPT’s financial reasoning is often shallow. It tends to fixate on simple linear trends and struggles to identify complex trend reversals or weigh conflicting signals between price data and sentiment.
The model often treats numerical and textual data as a single sequence rather than performing sophisticated cross-modal fusion. This lack of specialized attention limits its ability to correlate specific social media signals with precise price movements.
Conclusion
While ChatGPT demonstrates a baseline ability to process financial data and sentiment, it remains a "neophyte" compared to specialized multimodal models. Future work should focus on specialized fine-tuning (e.g., on financial corpora like BloombergGPT) and developing hybrid architectures that combine the linguistic power of LLMs with the quantitative precision of traditional financial models.
Summary updated