This paper, titled "Thinking Fast, Trading Late," provides a sobering reality check for the burgeoning field of using Large Language Models (LLMs) to predict stock market movements. While many studies claim LLMs can "beat the market" based on historical news, the authors argue that most of these gains are mirages created by look-ahead bias, memorization, and the simple fact that markets move faster than traders can read.
TL;DR
The authors introduce OLIVIA, a rigorous protocol to prevent backtesting bias, and apply it to AI decision models reading S&P 500 news. They find that while LLMs can predict overnight price moves, these moves are already "priced in" by the time a trader can act at the market open, and no tested strategy survives real-world trading costs.
What Problem Does This Paper Address?
The core challenge is the "backtesting delusion." In empirical finance, it is notoriously easy to find a signal that would have worked in the past but fails in the future. LLMs introduce specific risks:
- Look-ahead Bias: Models may have been trained on data that includes the "future" relative to the test period.
- Memorization: Even if a company name is masked, a model might recognize a specific event (e.g., a massive merger) and "remember" that the stock went up.
- Selection Bias: Researchers often try thousands of variations and only publish the one that happened to work.
Furthermore, the authors investigate market efficiency: even if a model correctly identifies news as "good," has the market already reacted by the time the news is public?
Methods
The authors propose the OLIVIA protocol, a set of six safeguards designed to make financial LLM research reproducible and honest:
- AI Decision Models: Instead of using generative LLMs (like ChatGPT) to write summaries, they use non-generative models that return specific probabilities for typed questions (e.g., "Will this stock rise by more than 1%?").
- Laya-Fin: An open-weights model (based on ModernBERT) fine-tuned specifically on 2024–2025 financial news and returns.
- Identity Masking: They replaced company names, tickers, and dates with generic placeholders to test if models were truly "reading" the news or just "remembering" the company.
- Decision Points: They tested models at three times: D0 (before the market opens), D1 (30 minutes after the open), and D2 (after the close).
Datasets
- Universe: Point-in-time membership of the S&P 500.
- News Feed: NewsWitch, a commercial feed providing titles and snippets with precise timestamps.
- Time Periods: Training (July 2024–June 2025), two validation quarters in 2025, and a blinded test period (January–July 2026).
Key Insights and Ideas
- Masking is not Anonymity: In a "matched-distractor" test, commercial models correctly identified the masked company in 62% to 74% of cases (against a 28% baseline). The models don't need the name; they recognize the unique "fingerprint" of a company's news (e.g., a specific drug trial result).
- The "Overnight" Trap: Commercial models showed a strong correlation with overnight returns (the move from yesterday's close to today's open). However, they showed almost zero ability to predict the intraday move (open to close). This suggests the market incorporates news into the price during pre-market trading, leaving nothing for a trader who waits for the official open.
- System One vs. System Two: Borrowing from psychology, the authors suggest LLMs act like "System One"—they can quickly categorize news as good or bad. However, they lack "System Two" reasoning—they cannot judge if the market overreacted or underreacted to that news.
- Validation Decay: A signal that looked highly significant in validation (IC of 0.065) dropped precipitously in the blinded test (IC of 0.020), highlighting how easily researchers can fool themselves without a "frozen" analysis plan.
Key Results
- Information Coefficient (IC): Commercial models achieved an IC of 0.074 to 0.077 for overnight moves, but this fell to near zero for moves occurring after the trader could actually enter a position.
- Economic Significance: No pre-registered trading strategy achieved a positive return after accounting for a modest 10 basis point (0.1%) trading cost.
- Variance Explanation: News readings explain only 0.25% to 0.82% of a stock's daily variance. The vast majority of stock price movement remains "noise" to these models.
Limitations and Future Work
- Large-Cap Bias: The study focused on the S&P 500. The authors acknowledge that news might diffuse more slowly in smaller, less-followed stocks, potentially leaving more room for LLM-based profits.
- Training Data Transparency: Because commercial models (like GPT-4 or Claude) have undisclosed training data, the authors could not fully rule out that the models had "seen" the test period during their pre-training.
- Future Forward Test: The authors have pre-registered a "forward test" to be run on news published after the models were released (late 2026/early 2027) to provide the cleanest possible test of memorization.
Summary updated