Introduction
The intersection of Large Language Models (LLMs) and finance has evolved from simple sentiment analysis toward autonomous "agentic" systems capable of making complex trading decisions. However, evaluating these agents has historically been hindered by proprietary datasets and narrow task focus. INVESTORBENCH addresses this by providing an open-source benchmark and framework designed to evaluate LLM agents across stocks, cryptocurrencies, and ETFs.
Financial markets are characterized by high noise, volatility, and multi-modal data. Existing research often lacks standardized evaluation grounds and versatile agent frameworks. INVESTORBENCH introduces a unified agent framework featuring a human-inspired Layered Long-Term Memory system, allowing models to navigate trading environments with a sophisticated understanding of information decay and relevance.
Methodology and Platform Architecture
The researchers modeled financial decision-making as a Partially Observable Markov Decision Process (POMDP). The proposed agent architecture consists of five core modules: Brain (the LLM), Perception, Profile (defining risk appetite), Memory, and Action.
The platform supports multiple financial decision-making tasks, including stock, cryptocurrency, and ETF trading. It integrates a multi-source market environment data warehouse containing financial news, SEC filings (10-K/Q), and market sentiment, powered by third-party APIs like Yahoo Finance and SEC EDGAR.
Layered Long-Term Memory
A standout innovation of INVESTORBENCH is the Layered Long-Term Memory module. Inspired by human cognition, it categorizes information into three tiers based on decay rates:
- Shallow Layer (14-day decay): Captures transient data like daily news and social media buzz.
- Intermediate Layer (90-day decay): Stores quarterly signals such as earnings reports.
- Deep Layer (365-day decay): Maintains foundational data like annual 10-K filings.
This tiered approach prevents agents from being overwhelmed by recent noise while ensuring they retain critical long-term structural data. The workflow involves observation, summarization, and reflection, with a retrieval mechanism that pulls essential memory events from each layer to inform decision-making.
Experimental Setup and Datasets
The benchmark utilizes three distinct environments:
- Stock Market: Covers 7 major equities (e.g., MSFT, AAPL, TSLA) using OHLCV data, SEC filings, and curated news.
- Cryptocurrency: Focuses on Bitcoin (BTC) and Ethereum (ETH), integrating price data with crypto-specific sentiment.
- ETF: Utilizes the News-Informed Financial Trend Yield (NIFTY) dataset.
The study evaluated 13 LLMs, ranging from proprietary models (GPT-4o, GPT-o1-preview) to open-source models (Llama 3.1, Qwen 2.5, DeepSeek) and financial domain-specific models (Palmyra-Fin-70B).
Key Insights and Results
The evaluation revealed several critical findings:
- Proprietary Dominance: GPT-4 and GPT-o1-preview consistently demonstrated superior reasoning and self-reflection. GPT-4o achieved an average Cumulative Return (CR) of 39.03% in stock trading, surpassing the "Buy & Hold" baseline of 34.10%.
- The 67B Parameter Threshold: For open-source models, a significant performance jump occurs at the 67B parameter mark. Models of this size, such as Qwen2.5-72B-Instruct, showed higher robustness and better Sharpe Ratios.
- Domain-Specific Limitations: Models fine-tuned on financial text (e.g., Palmyra-Fin-70B) excelled at report analysis but often struggled with the sequential logic required for active trading compared to general-purpose models.
- Memory Efficacy: The layered memory system proved more effective than standard similarity-based retrieval (RAG) by explicitly modeling information decay.
Conclusion and Future Work
INVESTORBENCH establishes a new standard for evaluating financial LLM agents. While current tasks focus on single-asset trading, future iterations aim to address multi-asset portfolio management and cross-asset hedging. Additionally, the researchers plan to incorporate more modalities, such as audio from earnings calls and visual data from technical charts, to further bridge the gap between AI agents and professional human investors.
Summary updated