LLM Trading Agents Earn Beta, Not Alpha: A Look-Ahead-Free Benchmark

Jackson WangarXiv 2026 · arXiv

TL;DR

A rigorous new benchmark, rta-bench-1, reveals that LLM trading agents don't actually possess stock-picking "skill" (alpha). Instead, their returns are almost entirely driven by market movements (beta) and their ability to reliably follow programmed risk-management rules, with performance often being indistinguishable from random stock selection within the same sector.

What Problem Does This Paper Address?

The financial world is currently flooded with "AI Trading Agents" that claim to beat the market. However, evaluating these agents is notoriously difficult due to three main "leaks":

  1. Look-Ahead Bias: Many backtests join news headlines to stock prices by calendar date. If a headline breaks at 6:00 PM, a model "deciding" at the 4:00 PM close shouldn't see it—but in many benchmarks, it does, effectively trading on the future.
  2. Data Contamination: LLMs are trained on massive historical datasets. If you ask a model to trade in 2022, it isn't "predicting" the bear market; it’s remembering it.
  3. The Luck Factor: Trading is noisy. The authors found that simply changing the random seed of a model can swing its returns by 5.2 percentage points—a margin larger than the difference between the "best" and "worst" models in some leaderboards.

The paper introduces rta-bench-1 to solve these issues by enforcing minute-by-minute news replay, using "blind" runs (hiding company names), and decomposing returns to see if the AI is actually smart or just riding a bull market.

Methods

The authors developed a backtesting harness that replays three distinct US market regimes: a rally (2026*), a bear market (2022), and a crash/rebound (2020).

Key methodological innovations include:

  • Publication-Time Replay: Headlines are only visible to the agent after their specific publication minute. Orders are filled at the next market open to ensure no "instantaneous" reaction to news that hasn't reached the exchange yet.
  • Factor Decomposition: Using a four-factor regression model (Market, Size, Growth, Momentum), the authors split returns into Beta (what the market gave the agent) and Alpha (what the agent's skill added).
  • Blind vs. Named Runs: Agents are tested both with real tickers and with aliases (e.g., "S01") to measure how much "memory" of historical events inflates their performance.
  • Look-Ahead Verification: A "stress test" where the system replays the agent's own replies on a market where the future has been randomized. If the agent still "wins," it's proof of a code leak.

Datasets

  • rta-bench-1: A new benchmark covering 56 large-cap US stocks across 11 sectors.
  • Market Periods: 2026 (Rally), 2022 (Bear), 2020 (COVID Crash/Rebound).
  • News Feed: A licensed feed of financial headlines and SEC Form 8-K filings, all timestamped to the minute.
  • Models Tested: Six models from four vendors, including GPT-5.6 Luna, Gemini 3.5 Flash, Claude (Haiku/Sonnet), and Llama 3.2 3B.

Key Insights and Ideas

  • The "News Edge" is an Illusion: When news is joined by calendar date, a simple rule (buy on beats, sell on misses) earns 32 basis points per trade. When replayed by publication minute, that same rule loses 6.7 basis points. The "edge" was just the model seeing the evening's news before the market closed.
  • LLMs as Executors, Not Analysts: While LLMs failed at picking winning stocks, they were excellent at following complex instructions. A prompt rule requiring 30% cash in "risk-off" markets successfully reduced drawdowns. The value of an LLM agent lies in its reliability as a rule-follower, not its predictive power.
  • The Sector Trap: Many agents appeared to pick "good" stocks, but when compared against a random selection of stocks from the same sector, the AI's advantage vanished. They weren't picking winners; they were just picking sectors that happened to be up.

Key Results

  • Zero Significant Alpha: Across 91 runs, not a single run produced a statistically significant four-factor alpha.
  • Beta Dominance: Market and style exposure accounted for a median of 78% of the total return.
  • Random Performance: The agents' decisions had a mean percentile rank of 54 (where 50 is pure chance).
  • Ablation Surprises: In tests where news was completely removed or swapped with news from the wrong company, the agents performed just as well. This suggests they weren't actually "reading" the news to find signals; they were likely reacting to price trends already in the prompt.

Limitations and Future Work

  • Large-Cap Focus: The study focused on the 56 largest US stocks. These are highly efficient markets where news is priced in within milliseconds. The authors acknowledge that LLMs might find more "alpha" in small-cap stocks where information travels slower.
  • General-Purpose Models: The study used off-the-shelf models. Specialized "Finance-tuned" models or agents with access to proprietary tools/data might perform differently.
  • Survivorship Bias: The stock list consists of companies that are large today, which inherently biases the historical backtest (though the factor regression accounts for this).

The authors conclude that for AI trading to be taken seriously, builders must provide the "edge" (data/signals) themselves and use LLMs primarily to execute the strategy, rather than expecting the model to "hallucinate" a winning trade from public headlines.

*Note: The paper uses "2026" as a test period, likely as a synthetic or future-dated dataset to ensure zero training data contamination.

Summary updated

Abstract

Large language model (LLM) trading agents now trade stocks for their users, and benchmarks rank models by their agents' returns. Yet a single run's return says little about skill. In our study, two runs of the same model that differed only in the random seed ended 5.2 points apart on average, so telling skill from luck takes many backtests across markets, models and seeds. However, live arenas trade each day only once, and backtests either use the few months after a model's training cutoff or run on years the model may remember, often with news joined to prices by calendar date. In this paper, we present rta-bench-1, a backtest over three US markets (a rally, a bear market and a crash). It shows each headline only after its publication minute, plays each period blind and named, and checks every run for look-ahead. It also splits each return into factor exposure and alpha and scores every decision against random choices of the same day and sector. In 91 runs of six models from four vendors, no alpha was significant, market and style exposure accounted for a median 78% of the return, and the decisions did no better than random stocks of the same sector. The agents bought the stocks with the most positive news, yet in ablations they earned as much without their headlines. Still, the agents carried out rules reliably, since a single cash rule in the prompt cut a GPT-based agent's share in stocks by 30.3 points in all six pairs. Builders should therefore supply the edge themselves and report exposures and alpha, and benchmarks should replay news by publication time.
BibTeX
@misc{wang2026llm,
  author = {Jackson Wang},
  title = {{LLM Trading Agents Earn Beta, Not Alpha: A Look-Ahead-Free Benchmark}},
  year = {2026},
  eprint = {2611.19530},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19530},
}
Figure