The State of the Art of Large Language Models on Chartered Financial Analyst Exams

Mahmoud Mahfouz, Ethan Callanan, Mathieu Sibue, Antony Papadimitriou, Zhiqiang Ma, Xiaomo Liu, Xiaodan ZhuEMNLP 2024 · DOI

TL;DR

This paper benchmarks the performance of 14 large language models (LLMs) on mock Chartered Financial Analyst (CFA) exams, revealing that proprietary models like GPT-40 and Claude 3 Opus lead in accuracy, particularly on multiple-choice questions. While LLMs can pass lower-level exams, they struggle significantly with the essay-based Level III, highlighting a critical gap in their ability to perform complex, real-world financial analysis and reasoning.

What Problem Does This Paper Address?

The Chartered Financial Analyst (CFA) program is a globally recognized credential for investment professionals, requiring extensive knowledge and analytical skills. As Large Language Models (LLMs) become increasingly capable and integrated into various industries, it's crucial to understand their performance on high-stakes professional exams. This paper addresses the gap in understanding how current state-of-the-art LLMs, both proprietary and open-source, perform on the CFA exams, which progressively test financial concepts, reasoning, and real-world application across three levels. The goal is to benchmark their capabilities, identify strengths and weaknesses, and guide practitioners on their suitability for financial industry automation.

Methods

The researchers evaluated 14 LLMs (5 proprietary, 9 open-source) on mock CFA exams for all three levels. They employed a one-shot learning with chain-of-thought (1S-CoT) prompting strategy, meaning each LLM received one example question with its reasoning and answer before being asked to solve a new question. For Level I and II multiple-choice questions (MCQs), they used the Accuracy metric. For Level III essay questions, a model-assisted human evaluation was conducted, where GPT-40 initially graded essays based on provided ground truth and grading details, followed by verification by a human CFA charterholder. To assess the impact of external knowledge, they also implemented a Retrieval-Augmented Generation (RAG) pipeline, allowing LLMs to access CFA textbooks.

Datasets

The study utilized mock CFA exams purchased from AnalystPrep for the 2023 curriculum, covering all three levels. These datasets included MCQs and essay questions, along with corresponding answers, explanations, and grading details. The researchers also leveraged CFA textbooks from the same provider for the RAG experiments.

Key Insights and Ideas

  • Proprietary models lead, but gaps remain: Leading proprietary LLMs like GPT-40 and Claude 3 Opus demonstrate the best overall performance, particularly on MCQs, but none can reliably pass the Level III exam, especially its essay component.
  • Essay writing is a major hurdle: The qualitative nature and complex reasoning required for Level III essay questions present a significant challenge for current LLMs, even for top-performing models.
  • Open-source models show promise: While generally underperforming proprietary models, open-source LLMs like Mixtral-8x22B and LLaMA 3 70B offer competitive capabilities for their size and cost, with LLaMA 3 models showing impressive financial reasoning.
  • RAG helps, but doesn't solve everything: Providing external theoretical knowledge via RAG can bridge the gap between open-source and proprietary models, especially for Level I, but its benefits diminish for higher levels due to increased abstraction and contextual reasoning demands.
  • Model size isn't everything: Architecture and training data quality appear more critical than sheer parameter count, as evidenced by the performance of models like LLaMA 3 70B and Mixtral-8x22B.
  • Prompting matters: The way LLMs are prompted significantly impacts their performance, with proprietary models showing a better ability to follow instructions precisely compared to open-source models.

Key Results

  • Level I & II Pass Rates: GPT-40 and Claude 3 Opus passed estimated minimum passing scores (60-70%) for Levels I and II. LLaMA 3 70B, with RAG, could also pass Level I and II at the lower bound.
  • Level III Failure: No LLM could reliably pass Level III. The best performing GPT-40 achieved an essay score of 46.2, bringing its overall Level III score to 55.0, below the estimated passing threshold.
  • Topic Performance: Proprietary models excelled in Portfolio Management on Level II. Open-source models generally struggled more with quantitative and financial reporting topics.
  • Cost: Proprietary models incurred higher costs for inference compared to open-source models, especially when considering the computational resources required.

Limitations and Future Work

The authors acknowledge that their essay grading method is not identical to the official CFA grading process, which could affect Level III results. They also note that the risk of data contamination for LLMs is reduced due to the proprietary nature of the mock exam data. Future work could focus on developing LLMs with improved essay writing capabilities, better contextual understanding, and more robust application of financial theory to real-world scenarios. Further research into domain-specific fine-tuning and advanced RAG techniques could also enhance LLM performance on such professional exams.

Summary updated

BibTeX
@inproceedings{mahfouz2024state,
  author = {Mahmoud Mahfouz and Ethan Callanan and Mathieu Sibue and Antony Papadimitriou and Zhiqiang Ma and Xiaomo Liu and Xiaodan Zhu},
  title = {{The State of the Art of Large Language Models on Chartered Financial Analyst Exams}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.80},
}
Figure