TL;DR
MTFinEval is a new benchmark designed to move beyond task-specific financial testing by evaluating the fundamental economic and management knowledge of Large Language Models (LLMs). Using 360 university-level questions across six disciplines, the researchers found that most current models—including top-tier open-source ones—struggle with these foundational principles, revealing a significant gap between task execution and theoretical understanding.
What Problem Does This Paper Address?
In the rush to deploy LLMs for stock trading, risk forecasting, and earnings analysis, we’ve relied heavily on "task-oriented" benchmarks. These benchmarks measure how well a model performs a specific job based on historical data. However, the authors argue that these datasets are "backward-looking" and increasingly unsuitable for real-world scenarios where policies shift or disruptive technologies (like AI itself) change the economic landscape.
The core problem is a lack of generalization. If a model doesn't understand the underlying economic theory—the "why" behind the data—it cannot reliably adapt to new situations. MTFinEval seeks to measure the "cognitive level" of an LLM’s theoretical knowledge to determine if it can be safely invested in production environments.
Methods
The researchers took a "back-to-basics" approach. Instead of scraping news or financial reports, they manually compiled 360 questions from university textbooks and major exam papers in economics and management.
To ensure the benchmark was rigorous:
- Expert Verification: They hired six economists to review every question for clarity and correctness, paying a bounty for each validated entry.
- Diverse Question Types: The benchmark includes single-choice, multiple-choice, and "judge" (true/false) questions.
- Zero-Shot Evaluation: Models were tested in a zero-shot environment (no prior examples) using specific prompts to act as a "financial knowledge expert."
- Mathematical Formulation: The authors defined objective functions to calculate the probability of the model selecting the correct answer string (e.g., "A, B, C") across different question formats.
Datasets
The primary contribution is the MTFinEval dataset. It is categorized into six major sub-disciplines:
- Macroeconomics
- Microeconomics
- Accounting
- Management
- E-commerce
- Strategic Management of Enterprise
The questions cover dimensions ranging from financial law and economic indicators to financial technology and situational adaptation. The paper also references existing benchmarks like BloombergGPT, FinMA, and EconNLI to contrast MTFinEval’s focus on foundational theory versus their focus on NLP tasks or specific reasoning.
Key Insights and Ideas
- The "Eurypalynous" Approach: The title uses the term "Eurypalynous" (borrowed from botany, meaning broad or diverse) to describe the wide variety of questions. The insight here is that an LLM’s financial utility isn't just about "finance"—it's about the intersection of accounting, law, and management.
- Theory as a Foundation: The authors posit that theoretical knowledge is the "basis for task execution." A model that fails at basic accounting principles is likely to hallucinate or fail when performing complex earnings analysis.
- Training Data Fingerprints: The results act as a "DNA test" for training data. For example, the Yi-1.5-9B model excelled in accounting and e-commerce, suggesting its training corpus was particularly rich in those specific domains.
- The Performance Gap: Perhaps the most striking insight is that all tested LLMs performed "poorly" overall. This suggests that while models are getting better at sounding like economists, they haven't yet mastered the undergraduate-level "first principles" of the field.
Key Results
The researchers evaluated several prominent open-source models released in 2024.
- Top Performer: Qwen2-7B-Instruct led the pack with a comprehensive score of 63.33%. It showed particular strength in accounting (80%).
- The Middle Ground: Models like GLM-4-9b-chat (58.06%) and Yi-1.5-9B-Chat (58.06%) showed respectable but inconsistent performance across sub-disciplines.
- The Underperformers: Llama-3-8B-Instruct and Qwen2-1.5B both struggled significantly, scoring only 40.0% overall.
- Domain Specificity: Gemma-2-9b-it showed a notable spike in Strategic Management (65%), while Yi-1.5-6B performed best in E-commerce (68.33%).
Limitations and Future Work
The authors acknowledge a few areas for growth:
- Scale: While 360 questions provide a high-quality "litmus test," larger datasets may be needed for more granular evaluation.
- Model Architecture: The paper notes that architectural limitations might hinder how models process and analyze economic information, suggesting that simply adding more data might not be enough.
- Future Directions: The authors suggest that future work should focus on increasing the "rigor reliability" of LLMs from the basics up, rather than just fine-tuning them on specific tasks. They advocate for models to be trained on more diverse, domain-specific theoretical data to bridge the current knowledge gap.
Summary updated