INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent

Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Koduvayur Subbalakshmi, Guojun Xiong, Jimin Huang, Lingfei Qian, Xueqing Peng, Qianqian Xie, Jordan W. Suchow

Introduction

The intersection of Large Language Models (LLMs) and finance has evolved from simple sentiment analysis toward autonomous "agentic" systems capable of making complex trading decisions. However, evaluating these agents has historically been hindered by proprietary datasets and narrow task focus. INVESTORBENCH addresses this by providing an open-source benchmark and framework designed to evaluate LLM agents across stocks, cryptocurrencies, and ETFs.

Financial markets are characterized by high noise, volatility, and multi-modal data. Existing research often lacks standardized evaluation grounds and versatile agent frameworks. INVESTORBENCH introduces a unified agent framework featuring a human-inspired Layered Long-Term Memory system, allowing models to navigate trading environments with a sophisticated understanding of information decay and relevance.

Methodology and Platform Architecture

The researchers modeled financial decision-making as a Partially Observable Markov Decision Process (POMDP). The proposed agent architecture consists of five core modules: Brain (the LLM), Perception, Profile (defining risk appetite), Memory, and Action.

Overview of the InvestorBench platform architecture, showcasing the integration of multi-source data, LLM tools, and diverse trading tasks.

The platform supports multiple financial decision-making tasks, including stock, cryptocurrency, and ETF trading. It integrates a multi-source market environment data warehouse containing financial news, SEC filings (10-K/Q), and market sentiment, powered by third-party APIs like Yahoo Finance and SEC EDGAR.

Layered Long-Term Memory

A standout innovation of INVESTORBENCH is the Layered Long-Term Memory module. Inspired by human cognition, it categorizes information into three tiers based on decay rates:

  • Shallow Layer (14-day decay): Captures transient data like daily news and social media buzz.
  • Intermediate Layer (90-day decay): Stores quarterly signals such as earnings reports.
  • Deep Layer (365-day decay): Maintains foundational data like annual 10-K filings.
Detailed workflow of the Memory Module, illustrating the interaction between working memory operations and the layered long-term memory layers.

This tiered approach prevents agents from being overwhelmed by recent noise while ensuring they retain critical long-term structural data. The workflow involves observation, summarization, and reflection, with a retrieval mechanism that pulls essential memory events from each layer to inform decision-making.

Experimental Setup and Datasets

The benchmark utilizes three distinct environments:

  1. Stock Market: Covers 7 major equities (e.g., MSFT, AAPL, TSLA) using OHLCV data, SEC filings, and curated news.
  2. Cryptocurrency: Focuses on Bitcoin (BTC) and Ethereum (ETH), integrating price data with crypto-specific sentiment.
  3. ETF: Utilizes the News-Informed Financial Trend Yield (NIFTY) dataset.

The study evaluated 13 LLMs, ranging from proprietary models (GPT-4o, GPT-o1-preview) to open-source models (Llama 3.1, Qwen 2.5, DeepSeek) and financial domain-specific models (Palmyra-Fin-70B).

Key Insights and Results

The evaluation revealed several critical findings:

  • Proprietary Dominance: GPT-4 and GPT-o1-preview consistently demonstrated superior reasoning and self-reflection. GPT-4o achieved an average Cumulative Return (CR) of 39.03% in stock trading, surpassing the "Buy & Hold" baseline of 34.10%.
  • The 67B Parameter Threshold: For open-source models, a significant performance jump occurs at the 67B parameter mark. Models of this size, such as Qwen2.5-72B-Instruct, showed higher robustness and better Sharpe Ratios.
  • Domain-Specific Limitations: Models fine-tuned on financial text (e.g., Palmyra-Fin-70B) excelled at report analysis but often struggled with the sequential logic required for active trading compared to general-purpose models.
  • Memory Efficacy: The layered memory system proved more effective than standard similarity-based retrieval (RAG) by explicitly modeling information decay.

Conclusion and Future Work

INVESTORBENCH establishes a new standard for evaluating financial LLM agents. While current tasks focus on single-asset trading, future iterations aim to address multi-asset portfolio management and cross-asset hedging. Additionally, the researchers plan to incorporate more modalities, such as audio from earnings calls and visual data from technical charts, to further bridge the gap between AI agents and professional human investors.

Summary updated

BibTeX
@inproceedings{li2025investorbench,
  author = {Haohang Li and Yupeng Cao and Yangyang Yu and Shashidhar Reddy Javaji and Zhiyang Deng and Yueru He and Yuechen Jiang and Zining Zhu and Koduvayur Subbalakshmi and Guojun Xiong and Jimin Huang and Lingfei Qian and Xueqing Peng and Qianqian Xie and Jordan W. Suchow},
  title = {{INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure