The Only Edge Was Memorized in Training: LLM Trading Agents Around Earnings and Bad News

Jackson WangarXiv 2026 · arXiv

In the world of quantitative finance, Large Language Models (LLMs) are often hailed as the next frontier for "alpha"—the elusive ability to beat the market. However, a new paper by Jackson Wang, titled "The Only Edge Was Memorized in Training: LLM Trading Agents Around Earnings and Bad News," delivers a sobering reality check. It suggests that the "intelligence" these agents display might just be a very expensive form of historical recall.

TL;DR

LLM trading agents fail to beat simple market rules when tested on data they haven't already seen. Their apparent success in trading earnings and "buying the dip" is largely driven by look-ahead bias—essentially "remembering" the past from their training data rather than reasoning about the future.

What Problem Does This Paper Address?

Financial news moves markets in minutes, often faster than an LLM can process a prompt and execute a trade. To find profit, traders look at two windows: before scheduled events (like earnings) and after major price shocks (bad news).

While many studies claim LLMs can trade these events successfully, they often suffer from look-ahead bias. If a model was trained in 2024, it already "knows" how Apple’s 2022 earnings turned out. Furthermore, many benchmarks don't account for trading costs or compare agents against "dumb" baselines, like simply buying every stock before an announcement. Wang’s paper seeks to isolate true reasoning from mere memorization.

Methods

The author tested five models from four vendors (including GPT, Gemini, Claude, and Llama) using a rigorous framework designed to kill look-ahead bias:

  1. The "Clock" Discipline: Every input was strictly cut off at the "decision close"—the last moment a trader could actually place a bet before the news broke.
  2. Named vs. Blind Testing: Agents were run twice. Once "Named" (with tickers and dates) and once "Blind" (where company names were replaced with aliases and dates were masked).
  3. Training Cutoff Split: Results were analyzed separately for events occurring before and after each model’s specific training cutoff date.
  4. Staged Reasoning: The study compared "single-step" agents (one prompt) against "multi-step" agents that reasoned through expectations, peer results, and insider filings before deciding.
Fig. 1. Overview of the two tests. Both replay the frozen data of rta-bench-1 (left). Before 598 earnings announcements and after 409 bad-news drops of 3% or more (middle), an agent decides from inputs published before the decision close, once with the stock’s name and once blind; the multi-step and four-call agents pass each answer to the next call, and a single call is the comparison. Every decision fills at the next open with trading costs and is compared with simple rules and a logistic regression on the same events (right), and every comparison is split at each model’s training cutoff (bottom).

Datasets

The study utilized rta-bench-1, a look-ahead-free benchmark containing:

  • 56 large US stocks across three distinct market regimes (2020, 2022, and 2026).
  • 598 earnings announcements and 409 "bad news" drops (price falls of 3% or more).
  • Clean Test Set: A subset of events from July 2026 onwards, guaranteed to be outside the training window of every model tested.

Key Insights and Ideas

  • The Memorization Trap: The most striking "aha moment" was that models were significantly more accurate on events that occurred before their training cutoff. When the ticker and date were known, models "predicted" directions they had already seen in their training sets.
  • The "Always Buy" Baseline: Before earnings, most agents simply defaulted to buying. While this made money in 2020, it wasn't due to LLM skill—it was simply capturing the "earnings announcement premium" that any simple rule could follow.
  • Reasoning vs. Regression: Surprisingly, adding complex reasoning steps (Chain of Thought) or extra data like insider filings didn't help. A simple logistic regression using the same numeric data performed as well as, or better than, the multi-billion parameter LLMs.
  • The Gemini Overreaction: In the "bad news" test, a Gemini-based agent judged almost every price drop as an "overreaction" and bought 96% of the dips. It didn't beat the market; it just followed a "buy everything" strategy.
Fig. 2. The clock of the two tests. Before earnings (top), the agent decides at the close of day −2 counted back from the reaction day, the order fills at the next open, and the position closes at the close of the reaction day, so a report before the open and a report after the close of the previous day land in the same position. After bad news (bottom), the agent decides at the close of the day the stock fell, a buy fills at the next open, and the position closes at the close of the 20th trading day.

Key Results

  • Vanishing Accuracy: A GPT-based model correctly called the direction of earnings 59.2% of the time for events before its cutoff, but this dropped to a near-random 53.3% for events after the cutoff.
  • Negative Alpha: After accounting for trading costs and market factors, no agent beat the simple baselines (like "always long" or a basic logistic regression) once the results were corrected for multiple comparisons.
  • Memory Probe: When given only a ticker and a date (and no news at all), models could still guess the direction of the stock move better than chance if the event was in their training set. This confirms that "named" backtests are essentially testing memory, not trading skill.
Comparison of agent performance against baselines; note how intervals often cross the zero line, indicating no significant edge.

Limitations and Future Work

The paper acknowledges a few key limitations:

  • Large-Cap Bias: The study focused on 56 large, surviving US stocks. Results might differ for small-cap stocks where news is less efficiently priced.
  • Survival Bias: The dataset only includes firms still listed today, which might make "buying the dip" look more attractive than it would be in a dataset including companies that went bankrupt.
  • Prompt Sensitivity: While the author tested various agent structures, the infinite space of "prompt engineering" means a better-prompted agent might exist, though these results suggest the bar is much higher than previously thought.

Wang concludes with a vital recommendation for the AI community: Always test trading agents "blind" and on data strictly following the training cutoff. Without these controls, you aren't measuring an agent's ability to trade; you're just measuring its ability to remember.

Summary updated

Abstract

Large language model (LLM) trading agents are tested on news, but news reaches stock prices within minutes, and a benchmark that replays headlines at their publication time finds no alpha in trading on them. Money can then remain in two places, before news whose date is known, such as earnings announcements, and after bad news, if the market's fall went too far. However, studies of LLMs and earnings read documents published with or after the announcement, and studies of reversal sort monthly portfolios, so neither tells whether an agent can trade these moments after costs. In this paper, we test LLM agents at both moments on the same look-ahead-free benchmark, before 598 earnings announcements and after 409 bad-news drops of large US stocks. Every input is cut at the decision close, staged agents are compared with single calls, simple rules and a logistic regression, and every decision is made with and without the stock's name. Across five models from four vendors, no agent beat the simple bets. Before earnings, the agents mostly bought and earned what buying before every announcement earned, and after bad news, a Gemini-based agent called almost every fall an overreaction while no agent picked better dips than buying them all. The only gains came from what the models had memorized in training. Given only the ticker and the dates, a GPT-based model called the direction of 59.2% of the announcements before its training cutoff and 53.3% after it. Agent builders should therefore compare an event agent with the simple bet and test it blind after the training cutoff.
BibTeX
@misc{wang2026only,
  author = {Jackson Wang},
  title = {{The Only Edge Was Memorized in Training: LLM Trading Agents Around Earnings and Bad News}},
  year = {2026},
  eprint = {2611.22177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22177},
}
Figure