TL;DR
A rigorous new benchmark, rta-bench-1, reveals that LLM trading agents don't actually possess stock-picking "skill" (alpha). Instead, their returns are almost entirely driven by market movements (beta) and their ability to reliably follow programmed risk-management rules, with performance often being indistinguishable from random stock selection within the same sector.
What Problem Does This Paper Address?
The financial world is currently flooded with "AI Trading Agents" that claim to beat the market. However, evaluating these agents is notoriously difficult due to three main "leaks":
- Look-Ahead Bias: Many backtests join news headlines to stock prices by calendar date. If a headline breaks at 6:00 PM, a model "deciding" at the 4:00 PM close shouldn't see it—but in many benchmarks, it does, effectively trading on the future.
- Data Contamination: LLMs are trained on massive historical datasets. If you ask a model to trade in 2022, it isn't "predicting" the bear market; it’s remembering it.
- The Luck Factor: Trading is noisy. The authors found that simply changing the random seed of a model can swing its returns by 5.2 percentage points—a margin larger than the difference between the "best" and "worst" models in some leaderboards.
The paper introduces rta-bench-1 to solve these issues by enforcing minute-by-minute news replay, using "blind" runs (hiding company names), and decomposing returns to see if the AI is actually smart or just riding a bull market.
Methods
The authors developed a backtesting harness that replays three distinct US market regimes: a rally (2026*), a bear market (2022), and a crash/rebound (2020).
Key methodological innovations include:
- Publication-Time Replay: Headlines are only visible to the agent after their specific publication minute. Orders are filled at the next market open to ensure no "instantaneous" reaction to news that hasn't reached the exchange yet.
- Factor Decomposition: Using a four-factor regression model (Market, Size, Growth, Momentum), the authors split returns into Beta (what the market gave the agent) and Alpha (what the agent's skill added).
- Blind vs. Named Runs: Agents are tested both with real tickers and with aliases (e.g., "S01") to measure how much "memory" of historical events inflates their performance.
- Look-Ahead Verification: A "stress test" where the system replays the agent's own replies on a market where the future has been randomized. If the agent still "wins," it's proof of a code leak.
Datasets
- rta-bench-1: A new benchmark covering 56 large-cap US stocks across 11 sectors.
- Market Periods: 2026 (Rally), 2022 (Bear), 2020 (COVID Crash/Rebound).
- News Feed: A licensed feed of financial headlines and SEC Form 8-K filings, all timestamped to the minute.
- Models Tested: Six models from four vendors, including GPT-5.6 Luna, Gemini 3.5 Flash, Claude (Haiku/Sonnet), and Llama 3.2 3B.
Key Insights and Ideas
- The "News Edge" is an Illusion: When news is joined by calendar date, a simple rule (buy on beats, sell on misses) earns 32 basis points per trade. When replayed by publication minute, that same rule loses 6.7 basis points. The "edge" was just the model seeing the evening's news before the market closed.
- LLMs as Executors, Not Analysts: While LLMs failed at picking winning stocks, they were excellent at following complex instructions. A prompt rule requiring 30% cash in "risk-off" markets successfully reduced drawdowns. The value of an LLM agent lies in its reliability as a rule-follower, not its predictive power.
- The Sector Trap: Many agents appeared to pick "good" stocks, but when compared against a random selection of stocks from the same sector, the AI's advantage vanished. They weren't picking winners; they were just picking sectors that happened to be up.
Key Results
- Zero Significant Alpha: Across 91 runs, not a single run produced a statistically significant four-factor alpha.
- Beta Dominance: Market and style exposure accounted for a median of 78% of the total return.
- Random Performance: The agents' decisions had a mean percentile rank of 54 (where 50 is pure chance).
- Ablation Surprises: In tests where news was completely removed or swapped with news from the wrong company, the agents performed just as well. This suggests they weren't actually "reading" the news to find signals; they were likely reacting to price trends already in the prompt.
Limitations and Future Work
- Large-Cap Focus: The study focused on the 56 largest US stocks. These are highly efficient markets where news is priced in within milliseconds. The authors acknowledge that LLMs might find more "alpha" in small-cap stocks where information travels slower.
- General-Purpose Models: The study used off-the-shelf models. Specialized "Finance-tuned" models or agents with access to proprietary tools/data might perform differently.
- Survivorship Bias: The stock list consists of companies that are large today, which inherently biases the historical backtest (though the factor regression accounts for this).
The authors conclude that for AI trading to be taken seriously, builders must provide the "edge" (data/signals) themselves and use LLMs primarily to execute the strategy, rather than expecting the model to "hallucinate" a winning trade from public headlines.
*Note: The paper uses "2026" as a test period, likely as a synthetic or future-dated dataset to ensure zero training data contamination.
Summary updated