TL;DR
This paper investigates whether Large Language Model (LLM) trading agents actually "understand" financial news or simply echo market moves that have already occurred. By building a multi-agent system with mathematically proven "influence bounds," the authors demonstrate that while LLMs can accurately categorize news, the vast majority of the "predictive" signal in web news is actually a lagging indicator of price moves that happened before the news was even published.
What Problem Does This Paper Address?
The rise of LLM-based trading agents has sparked a debate: are these models truly uncovering "alpha" from text, or are they victims of look-ahead bias?
Financial news on the web is often "reactive"—it reports on a price move that has already happened. If an LLM reads a headline saying "Stock X jumps 5% on earnings beat" and predicts the stock will go up, it isn't predicting the future; it's observing the past. The authors tackle three major confounds:
- Timing: The reported event often precedes the news item.
- Memorization: The model might have seen the event in its training data.
- Content: The news item itself might explicitly mention the price move.
The core question is: Do LLM judgments track the market's reaction to an event, or do they merely follow it?
Methods
The authors designed a multi-agent trading system (Figure 1) built on a "blackboard architecture" where agents (News, Policy, Technical, etc.) share write-once, point-in-time data stores but do not exchange free-text messages.
Key technical innovations include:
- Typed Question Programs (Jev Schema): Instead of generating free text, models return a probability distribution over a fixed set of options (e.g., "clearly harmful" to "clearly helpful"). This prevents "prompt injection" attacks from leaking into the trading logic.
- Influence Bounds: The system uses a "trust-partitioned" learner. A mathematical bound ensures that untrusted web text can only move a stock's position within a specific "band" defined by trusted data (like prices and SEC filings). This limits the damage a "planted" fake news item can do.
- Decision Cards: Every trade is logged with a "card" that traces the exact contribution of each agent and news item to the final decision, enabling perfect attribution.
- Confirmatory Reader ("emu"): A pinned version of GPT-4o-mini was used to serve the schema, tested against a "validity harness" that compares judgments to actual market returns and official SEC 8-K filings.
Datasets
- Universe: The 100 most-traded S&P 500 members.
- News: 100,906 items from Benzinga (via Alpaca's API) covering a 115-week evaluation window.
- Ground Truth: Official SEC 8-K filings (mandatory reports of material events) used to verify if the LLM correctly identified the type of event reported.
- Market Data: Prices and S&P 500 ETF (SPY) returns for calculating abnormal returns (AR).
Key Insights and Ideas
- The "Lag" is the Signal: The study found that 72% of earnings news items appear after the issuer's official SEC filing. By the time the news hits the web, the market has usually already priced it in.
- Reporting the Move: Contributor-written news often explicitly reports the price move itself. When these "price-reporting" items are removed, the correlation between LLM judgments and market returns drops significantly.
- Validity vs. Usefulness: An LLM can be a "valid" reader (correctly identifying that news is "good") without being a "useful" trader. If the "good" news is published after the price jump, the trader is too late.
- Steerability and Safety: The "influence bound" proved highly effective. Even under a simulated "attack" where fake news was planted, the system's logic prevented the portfolio from being hijacked, while still allowing legitimate news to influence trades within safe limits.
Key Results
- Tracking the Event: LLM judgments showed a weekly Spearman correlation of 0.119 with announcement-window abnormal returns.
- The Publication Drop-off: When measured from the first trade after publication (the actual time a trader could act), the correlation plummeted to 0.023.
- 8-K Agreement: The models were excellent at categorizing events, achieving an AUC of 0.833 when matching news items to official SEC 8-K filing codes.
- Attack Robustness: Under attack, "decision cards" successfully traced 88% of moved positions back to the specific planted item, proving the system's auditability.
- Look-ahead Control: The association held even for news published after the LLM's knowledge cutoff, suggesting the results aren't just due to the model "remembering" the past.
Limitations and Future Work
- Weekly Horizon: The system operates on a weekly rebalancing cycle. The authors note that the "alpha" from news might be assimilated so fast (within minutes) that a weekly system cannot capture it, even if the reader is valid.
- Universe Size: The study focused only on the 100 largest S&P 500 firms. Smaller, less-covered firms might have news that lags the market less significantly.
- Future Directions: The authors propose that future "market-graded" tests of LLMs should report results from the time of publication, not just the time of the event, to provide a realistic measure of utility.
Summary updated