Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams

Ethan Callanan, Amarachi Mbakwe, Antony Papadimitriou, Yulong Pei, Mathieu Sibue, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, Sameena ShaharXiv 2023 · DOI · arXiv

Introduction

The Chartered Financial Analyst (CFA) credential is one of the most respected and rigorous certifications in the global finance industry. Achieving it requires passing three levels of exams that test a candidate's mastery of investment tools, asset valuation, portfolio management, and ethical standards. As Large Language Models (LLMs) like ChatGPT and GPT-4 demonstrate increasing proficiency in general natural language tasks, a critical question arises: can these models handle the specialized, multi-step reasoning required of a professional financial analyst?

This paper evaluates the performance of OpenAI’s ChatGPT (gpt-3.5-turbo) and GPT-4 on mock CFA Level I and Level II exams. The study explores whether "out-of-the-box" LLMs possess the necessary financial literacy and reasoning capabilities to match human professionals, identifying specific areas of strength and persistent failure modes.

Methodology and Datasets

The researchers employed three primary prompting strategies to assess the models:

  1. Zero-Shot (ZS): Direct questioning without prior examples.
  2. Chain-of-Thought (CoT): Encouraging the model to "think step-by-step" to decompose complex logic.
  3. Few-Shot (FS): Providing between 2 and 10 examples (e.g., 2S, 10S) to establish context.

The evaluation utilized a dataset of mock CFA exams, comprising five exams for Level I and two for Level II. Level I consists of 180 independent multiple-choice questions (MCQs) focusing on fundamental concepts. Level II introduces "item sets" or "vignettes"—lengthy case descriptions accompanied by data tables—followed by several related MCQs. This structure is designed to simulate real-world financial analysis, where relevant information must be extracted from a sea of narrative and numerical data.

Performance Analysis

The results indicate a significant performance gap between the two models and between the two exam levels. GPT-4 emerged as the superior model, achieving scores that likely surpass the estimated human passing threshold (typically 60-70%).

Performance of ChatGPT and GPT-4 across different prompting settings for CFA Level I and Level II.

As shown in the table above, GPT-4 reached a peak accuracy of 74.6% on Level I and 61.4% on Level II. In contrast, ChatGPT performed reasonably well on Level I (63.0%) but struggled significantly with Level II, plummeting to 47.6%. This decline highlights the "vignette" difficulty spike. Level II prompts are approximately ten times longer than Level I prompts, leading to "context dilution" where the model fails to filter irrelevant details from the dense narratives.

Error Analysis and the "CoT Paradox"

A qualitative analysis of the models' errors reveals distinct failure modes. For ChatGPT, the primary hurdle was "knowledge-based"—it often lacked the specific financial formulas or concepts required. For GPT-4, errors were more frequently "calculation-based," where the model understood the logic but failed the arithmetic.

Distribution of error types for ChatGPT and GPT-4.

The study also uncovered a "Chain-of-Thought (CoT) Paradox." While CoT is generally expected to improve reasoning, it actually decreased ChatGPT's performance on Level I. The model frequently "talked itself" into incorrect conclusions by hallucinating formulas or making logical leaps that deviated from the correct path. However, CoT proved beneficial for Level II, where the structured approach helped models navigate the complexity of the vignettes. For ChatGPT, the "sweet spot" for Level I was often a simple two-shot (2S) prompt, which provided enough context without the risks of long-form reasoning.

Conclusion and Future Directions

While GPT-4 shows promise as a tool for financial analysis, several limitations remain. The models are currently "mathematically fragile," often hallucinating calculations. Future iterations could benefit from offloading arithmetic to specialized tools like Wolfram Alpha. Furthermore, the reliance on static training data suggests that Retrieval-Augmented Generation (RAG) will be essential for accessing real-time market data and evolving regulations.

The researchers also propose a "critic" architecture, where one model generates a response and a second model reviews it to identify logical inconsistencies or hallucinations. As LLMs continue to evolve, bridging the gap between general literacy and specialized professional reasoning will be the next frontier in AI-driven finance.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance on a wide range of Natural Language Processing (NLP) tasks, often matching or even beating state-of-the-art task-specific models. This study aims at assessing the financial reasoning capabilities of LLMs. We leverage mock exam questions of the Chartered Financial Analyst (CFA) Program to conduct a comprehensive evaluation of ChatGPT and GPT-4 in financial analysis, considering Zero-Shot (ZS), Chain-of-Thought (CoT), and Few-Shot (FS) scenarios. We present an in-depth analysis of the models' performance and limitations, and estimate whether they would have a chance at passing the CFA exams. Finally, we outline insights into potential strategies and improvements to enhance the applicability of LLMs in finance. In this perspective, we hope this work paves the way for future studies to continue enhancing LLMs for financial reasoning through rigorous evaluation.
BibTeX
@misc{callanan2023can,
  author = {Ethan Callanan and Amarachi Mbakwe and Antony Papadimitriou and Yulong Pei and Mathieu Sibue and Xiaodan Zhu and Zhiqiang Ma and Xiaomo Liu and Sameena Shah},
  title = {{Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams}},
  year = {2023},
  eprint = {2310.08678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.08678},
}
Figure