Abstract
Financial question answering (QA) presents unique challenges for large language models (LLMs) due to the requirement for multi-step quantitative reasoning and domain-specific expertise. We present a role-aware multi-agent framework designed to enhance LLM performance in financial education. By decomposing the QA process into specialized roles—Evidence Retriever, Base Generator, and Expert Reviewer—the system provides a structured mechanism for retrieval-augmented generation and iterative critique. Our evaluation on a dataset of 3,532 expert-designed questions demonstrates that this framework allows smaller models to match the performance of specialized, fine-tuned financial models, with the expert critique mechanism providing the most significant accuracy gains.
Introduction
Financial QA is notoriously difficult for general-purpose AI. Unlike basic mathematics, professional finance requires mastery of specialized terminology (e.g., "WACC," "payout ratios") and the ability to parse unstructured data from financial statements and tax regulations. While LLMs have shown promise in general reasoning, they frequently suffer from hallucinations regarding financial facts or fail to follow the rigorous procedural steps required for professional certifications like the CFA or CPA.
Existing solutions, such as fine-tuning models on financial corpora, are resource-intensive and often lack the "critique" mechanism necessary to identify subtle arithmetic or conceptual errors. This paper addresses these gaps by proposing a multi-agent architecture that simulates a professional workflow.
Methodology: The Role-Aware Multi-Agent System
The proposed framework moves beyond simple Chain-of-Thought (CoT) prompting by employing three distinct agents in a single-pass iteration:
- Evidence Retriever (ER): This agent acts as an "open-book" researcher. Using Retrieval-Augmented Generation (RAG), it extracts relevant facts, formulas, and definitions from six authoritative finance textbooks to ground the subsequent reasoning process.
- Base Generator (BG): Serving as the "drafter," this agent utilizes the retrieved evidence and the original question to produce an initial answer using step-by-step reasoning.
- Expert Reviewer (XR): Acting as an "auditor," this agent is assigned a specific persona (e.g., "Tax Analyst" or "Equity Expert") based on the question's topic. It audits the draft for logical inconsistencies, formula errors, and conceptual misunderstandings, providing a final critique to refine the output.
The researchers evaluated four configurations: M-0 (standard closed-book baseline), M-1 (adding evidence), M-2 (adding critique), and M-3 (the full system combining both).
Experimental Setup and Datasets
The framework was evaluated using a robust dataset of 3,532 expert-designed finance questions sourced from Study.com. These questions span 82 subtopics categorized into seven major domains: Investments & Valuation, Income & Interest, Financial Statements & Analysis, Derivatives & Risk Management, Corporate Finance & Capital Management, Taxation & Payroll, and Budgeting & Personal Finance. To ground the agents, the authors indexed six foundational finance textbooks, including Principles of Corporate Finance by Brealey et al., to serve as the RAG knowledge base.
Results and Key Insights
The full framework (M-3) consistently outperformed the zero-shot baseline across all tested models. Gemini-2.0-Flash emerged as the top performer, reaching 79.14% accuracy, an 8.27% gain over its baseline. Notably, GPT-4o-mini reached 69.93% accuracy, effectively "leveling up" to match specialized finance models like FinGPT-mt.
Several key insights emerged from the data:
- Critique Outperforms Retrieval: Surprisingly, the Expert Reviewer (critique) provided a larger performance boost than the Evidence Retriever (RAG). While RAG grounds the model in facts, the reviewer agent is more effective at catching procedural and arithmetic mistakes.
- The Power of Personas: Assigning specific roles (e.g., "You are a derivatives specialist") steers the model toward domain-specific reasoning paths that generic prompts often miss.
- The "Everyday Language" Trap: Models struggled most with "Budgeting & Personal Finance." This category mixes casual language with strict financial rules, making it difficult for the AI to translate context into precise mathematical problems.
Conclusion and Future Work
The role-aware multi-agent framework demonstrates that structured collaboration between specialized agents can significantly improve LLM accuracy in complex domains. While the approach increases latency and cost due to multiple LLM calls, it provides a path for smaller models to achieve professional-grade results.
Future research will focus on integrating external tools, such as Python-based calculators or table parsers, to further reduce arithmetic errors. Additionally, the authors plan to investigate how the specific wording and structure of the "critique" impact final outputs and expand testing to a broader range of financial data sources beyond textbooks.