Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment

Shuoling Liu, Gaoguo Jia, Yuhang Jiang, Liyuan Chen, Qiang YangarXiv 2024 · DOI · arXiv

Introduction

In behavioral finance, the framing effect describes a cognitive bias where individuals make different decisions based on how information is presented rather than the objective facts. This bias is a significant hurdle for human investors and, as recent research shows, for Large Language Models (LLMs) like ChatGPT. When tasked with financial sentiment analysis, LLMs often mirror human reactive tendencies, focusing on short-term sensationalism rather than long-term economic indicators.

This paper introduces the Classify-and-Rethink (CAR) framework, a multi-step prompting strategy designed to mitigate these biases. By structuring the reasoning process, the CAR framework enables LLMs to function as more rational, long-term investors, significantly improving performance in gold market trading.

The Classify-and-Rethink (CAR) Framework

The CAR framework moves away from simple "one-step" sentiment scoring. Instead, it employs a zero-shot reasoning chain that mimics professional financial analysis:

  1. Classify: The model first categorizes news into one of six domains (e.g., Geopolitical events, Macroeconomic data, Central Bank policy). This step forces the model to identify the underlying nature of the information before judging its impact.
  2. Initial Scoring: Based on the identified category, the model provides an initial bullish or bearish score ranging from -1 to 1.
  3. Rethink: This critical step prompts the model to adopt a "reverse thinking perspective." It reflects on whether the initial score is reasonable within a long-term trend context, making necessary adjustments to produce a final, balanced score.
The multi-step workflow of the Classify-and-Rethink (CAR) framework, from text input to investment decision.

Following the scoring process, a trading strategy is applied by calculating a moving average of the scores from the 20 most recent news articles. If the average crosses a specific threshold, it triggers a "Buy" (full position) or "Sell" (empty position) signal.

Experimental Setup and Data

The study evaluated the CAR framework using data from January 2, 2018, to June 13, 2023. The dataset included:

  • Price Data: Daily closing prices of the Shanghai Gold Exchange spot gold index (Au9999.SGE).
  • News Data: Chinese-language gold market news from dyhjw.com.

Traditional sentiment models often struggle with this news data, frequently labeling it as "neutral" due to a lack of overt emotional language. This makes it an ideal testbed for the sophisticated reasoning capabilities of LLMs.

Key Insights: The Logic of Distribution

A primary finding of the study was the impact of the CAR framework on the distribution of sentiment scores. Simple "One-Step" prompts often resulted in heavily skewed distributions. While the "Classify" step alone tended to push scores toward extremes, the full CAR strategy produced a near-normal distribution. This distribution is more representative of actual market dynamics, where extreme events are rarer than moderate fluctuations.

Ablation studies further highlighted the importance of the classification step. Removing it caused the strategy's return to drop to 42%—underperforming even a basic buy-and-hold approach. This suggests that identifying the type of news is a prerequisite for accurate financial reasoning.

Performance Evaluation

The CAR framework demonstrated clear superiority over both traditional benchmarks and simpler AI prompting methods. In back-testing, the CAR strategy achieved an 80.35% return with a Sharpe Ratio of 1.071.

Comparative performance of different investment strategies, showing the Net Asset Value (NAV) over time.

As shown in the performance comparison, the CAR strategy was particularly effective at tracking abrupt gold price increases and avoiding the significant drawdowns that plagued simpler "Classify-only" or sentiment-based methods.

Strategy Return Sharpe Ratio
Buy-and-Hold 63.53% 0.811
One-Step (Simple Sentiment) 63.44% 0.902
Classify (Two-Step) 73.41% 1.019
Classify+Rethink (CAR) 80.35% 1.071

Conclusion and Future Work

The research demonstrates that structured prompting can effectively mitigate behavioral biases in LLMs, turning them into more reliable tools for financial decision-making. However, several challenges remain. The "alpha" or excess returns generated by such strategies may diminish as LLM-based trading becomes more widespread. Additionally, the method requires a high "reasoning floor," as smaller open-source models often struggle with the complex numerical scoring required.

Future research will explore more sophisticated portfolio management techniques, dynamic thresholding, and the application of this framework across different asset classes and languages.

Summary updated

BibTeX
@misc{liu2024can,
  author = {Shuoling Liu and Gaoguo Jia and Yuhang Jiang and Liyuan Chen and Qiang Yang},
  title = {{Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment}},
  year = {2024},
  eprint = {2411.13599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.13599},
}
Figure