Introduction
The Chartered Financial Analyst (CFA) credential is one of the most respected and rigorous certifications in the global finance industry. Achieving it requires passing three levels of exams that test a candidate's mastery of investment tools, asset valuation, portfolio management, and ethical standards. As Large Language Models (LLMs) like ChatGPT and GPT-4 demonstrate increasing proficiency in general natural language tasks, a critical question arises: can these models handle the specialized, multi-step reasoning required of a professional financial analyst?
This paper evaluates the performance of OpenAI’s ChatGPT (gpt-3.5-turbo) and GPT-4 on mock CFA Level I and Level II exams. The study explores whether "out-of-the-box" LLMs possess the necessary financial literacy and reasoning capabilities to match human professionals, identifying specific areas of strength and persistent failure modes.
Methodology and Datasets
The researchers employed three primary prompting strategies to assess the models:
- Zero-Shot (ZS): Direct questioning without prior examples.
- Chain-of-Thought (CoT): Encouraging the model to "think step-by-step" to decompose complex logic.
- Few-Shot (FS): Providing between 2 and 10 examples (e.g., 2S, 10S) to establish context.
The evaluation utilized a dataset of mock CFA exams, comprising five exams for Level I and two for Level II. Level I consists of 180 independent multiple-choice questions (MCQs) focusing on fundamental concepts. Level II introduces "item sets" or "vignettes"—lengthy case descriptions accompanied by data tables—followed by several related MCQs. This structure is designed to simulate real-world financial analysis, where relevant information must be extracted from a sea of narrative and numerical data.
Performance Analysis
The results indicate a significant performance gap between the two models and between the two exam levels. GPT-4 emerged as the superior model, achieving scores that likely surpass the estimated human passing threshold (typically 60-70%).
As shown in the table above, GPT-4 reached a peak accuracy of 74.6% on Level I and 61.4% on Level II. In contrast, ChatGPT performed reasonably well on Level I (63.0%) but struggled significantly with Level II, plummeting to 47.6%. This decline highlights the "vignette" difficulty spike. Level II prompts are approximately ten times longer than Level I prompts, leading to "context dilution" where the model fails to filter irrelevant details from the dense narratives.
Error Analysis and the "CoT Paradox"
A qualitative analysis of the models' errors reveals distinct failure modes. For ChatGPT, the primary hurdle was "knowledge-based"—it often lacked the specific financial formulas or concepts required. For GPT-4, errors were more frequently "calculation-based," where the model understood the logic but failed the arithmetic.
The study also uncovered a "Chain-of-Thought (CoT) Paradox." While CoT is generally expected to improve reasoning, it actually decreased ChatGPT's performance on Level I. The model frequently "talked itself" into incorrect conclusions by hallucinating formulas or making logical leaps that deviated from the correct path. However, CoT proved beneficial for Level II, where the structured approach helped models navigate the complexity of the vignettes. For ChatGPT, the "sweet spot" for Level I was often a simple two-shot (2S) prompt, which provided enough context without the risks of long-form reasoning.
Conclusion and Future Directions
While GPT-4 shows promise as a tool for financial analysis, several limitations remain. The models are currently "mathematically fragile," often hallucinating calculations. Future iterations could benefit from offloading arithmetic to specialized tools like Wolfram Alpha. Furthermore, the reliance on static training data suggests that Retrieval-Augmented Generation (RAG) will be essential for accessing real-time market data and evolving regulations.
The researchers also propose a "critic" architecture, where one model generates a response and a second model reviews it to identify logical inconsistencies or hallucinations. As LLMs continue to evolve, bridging the gap between general literacy and specialized professional reasoning will be the next frontier in AI-driven finance.
Summary updated