In the world of quantitative finance, Large Language Models (LLMs) are often hailed as the next frontier for "alpha"—the elusive ability to beat the market. However, a new paper by Jackson Wang, titled "The Only Edge Was Memorized in Training: LLM Trading Agents Around Earnings and Bad News," delivers a sobering reality check. It suggests that the "intelligence" these agents display might just be a very expensive form of historical recall.
TL;DR
LLM trading agents fail to beat simple market rules when tested on data they haven't already seen. Their apparent success in trading earnings and "buying the dip" is largely driven by look-ahead bias—essentially "remembering" the past from their training data rather than reasoning about the future.
What Problem Does This Paper Address?
Financial news moves markets in minutes, often faster than an LLM can process a prompt and execute a trade. To find profit, traders look at two windows: before scheduled events (like earnings) and after major price shocks (bad news).
While many studies claim LLMs can trade these events successfully, they often suffer from look-ahead bias. If a model was trained in 2024, it already "knows" how Apple’s 2022 earnings turned out. Furthermore, many benchmarks don't account for trading costs or compare agents against "dumb" baselines, like simply buying every stock before an announcement. Wang’s paper seeks to isolate true reasoning from mere memorization.
Methods
The author tested five models from four vendors (including GPT, Gemini, Claude, and Llama) using a rigorous framework designed to kill look-ahead bias:
- The "Clock" Discipline: Every input was strictly cut off at the "decision close"—the last moment a trader could actually place a bet before the news broke.
- Named vs. Blind Testing: Agents were run twice. Once "Named" (with tickers and dates) and once "Blind" (where company names were replaced with aliases and dates were masked).
- Training Cutoff Split: Results were analyzed separately for events occurring before and after each model’s specific training cutoff date.
- Staged Reasoning: The study compared "single-step" agents (one prompt) against "multi-step" agents that reasoned through expectations, peer results, and insider filings before deciding.
Datasets
The study utilized rta-bench-1, a look-ahead-free benchmark containing:
- 56 large US stocks across three distinct market regimes (2020, 2022, and 2026).
- 598 earnings announcements and 409 "bad news" drops (price falls of 3% or more).
- Clean Test Set: A subset of events from July 2026 onwards, guaranteed to be outside the training window of every model tested.
Key Insights and Ideas
- The Memorization Trap: The most striking "aha moment" was that models were significantly more accurate on events that occurred before their training cutoff. When the ticker and date were known, models "predicted" directions they had already seen in their training sets.
- The "Always Buy" Baseline: Before earnings, most agents simply defaulted to buying. While this made money in 2020, it wasn't due to LLM skill—it was simply capturing the "earnings announcement premium" that any simple rule could follow.
- Reasoning vs. Regression: Surprisingly, adding complex reasoning steps (Chain of Thought) or extra data like insider filings didn't help. A simple logistic regression using the same numeric data performed as well as, or better than, the multi-billion parameter LLMs.
- The Gemini Overreaction: In the "bad news" test, a Gemini-based agent judged almost every price drop as an "overreaction" and bought 96% of the dips. It didn't beat the market; it just followed a "buy everything" strategy.
Key Results
- Vanishing Accuracy: A GPT-based model correctly called the direction of earnings 59.2% of the time for events before its cutoff, but this dropped to a near-random 53.3% for events after the cutoff.
- Negative Alpha: After accounting for trading costs and market factors, no agent beat the simple baselines (like "always long" or a basic logistic regression) once the results were corrected for multiple comparisons.
- Memory Probe: When given only a ticker and a date (and no news at all), models could still guess the direction of the stock move better than chance if the event was in their training set. This confirms that "named" backtests are essentially testing memory, not trading skill.
Limitations and Future Work
The paper acknowledges a few key limitations:
- Large-Cap Bias: The study focused on 56 large, surviving US stocks. Results might differ for small-cap stocks where news is less efficiently priced.
- Survival Bias: The dataset only includes firms still listed today, which might make "buying the dip" look more attractive than it would be in a dataset including companies that went bankrupt.
- Prompt Sensitivity: While the author tested various agent structures, the infinite space of "prompt engineering" means a better-prompted agent might exist, though these results suggest the bar is much higher than previously thought.
Wang concludes with a vital recommendation for the AI community: Always test trading agents "blind" and on data strictly following the training cutoff. Without these controls, you aren't measuring an agent's ability to trade; you're just measuring its ability to remember.
Summary updated