TL;DR
The paper introduces PROMOTE, a rigorous protocol for financial research that prevents "backtest overfitting" by treating strategy discovery as a budget-constrained betting game. By using anytime-valid e-processes, it allows researchers and AI agents to continuously monitor and add new trading rules without inflating false-positive rates, ensuring that "promoted" strategies have genuine statistical evidence rather than just being lucky.
What Problem Does This Paper Address?
In quantitative finance, the "backtest" is the standard way to vet a trading strategy. However, traditional backtesting is plagued by data snooping and optional stopping. A researcher (or an autonomous AI agent) rarely runs just one test; they tweak parameters, change the asset universe, or drop unpromising rules based on interim results.
Treating the final "winning" rule as if it came from a single, fixed test is a statistical fallacy. At machine speeds, AI agents can generate thousands of variations, making it almost certain that some will look profitable purely by chance. This leads to "overfitted" strategies that perform brilliantly in the past but fail immediately upon deployment. The authors seek a way to authorize continuous, adaptive research while maintaining a strict guarantee on the probability of promoting a "dud" strategy.
Methods
The authors propose a protocol called PROMOTE, built on the foundations of anytime-valid inference and e-processes.
- Statistical Wealth Budget: The research program starts with a total "wealth" of 1. Every time a new candidate strategy $h$ is created, it is assigned a portion of this wealth ($\pi_h$), such that the sum of all assigned wealth never exceeds 1.
- Predictable Birth: A strategy must be "born" at a specific time. It cannot "inherit" evidence from before its birth; it only starts accumulating evidence from returns realized after it is registered.
- E-Processes as Evidence: For each strategy, the protocol tracks an e-process—a non-negative supermartingale that represents the "winnings" from betting against the null hypothesis (that the strategy has no edge).
- The Promotion Threshold: A strategy is promoted only if its e-process crosses a boundary of $1/(\alpha \pi_h)$, where $\alpha$ is the desired false-promotion rate (e.g., 5%).
- Cost-Awareness: The protocol explicitly includes transaction costs and uses "clipped" returns to ensure the test remains valid even if the underlying return distribution has heavy tails.
This approach turns the research process into an auditable ledger. You can add as many strategies as you want, and you can stop whenever you see a crossing, provided you have the "statistical budget" to fund them.
Datasets
The authors evaluate PROMOTE using three distinct setups:
- Shared-Sign Null Paths: 3,000 Monte Carlo simulations using randomized signs of real historical returns to create a "null" environment where no strategy has a true edge.
- Exact Mean-Zero Experiment: 3,000 simulated research programs where an agent selects candidate identities and birth times based on past performance, testing the protocol's resilience to adaptive selection.
- Public Market Replay: A "pseudo-out-of-sample" replay using real-world data from March 2006 to May 2026 (including projected dates). This includes 24 hypotheses across equity ETFs (SPY, QQQ, etc.), defensive ETFs (IEF, GLD), and Fama-French industry portfolios.
Key Insights and Ideas
- Research as a Ledger: The most important takeaway is that research is a process, not a single event. PROMOTE formalizes this by requiring an append-only ledger of every idea, including the ones that failed.
- The Price of Validity: Validity isn't free. The protocol shows that to be "anytime-valid," you must wait longer for evidence to accumulate. A "lucky" naive test might trigger early, but PROMOTE requires persistent evidence to cross its higher, wealth-adjusted boundary.
- Independence is Not Required: Unlike many multi-comparison corrections (like Bonferroni), this method handles dependent candidates (e.g., two strategies that trade similar assets) without losing its safety guarantee.
- Separation of Discovery and Deployment: A "promotion" is not a guarantee of future profit; it is a certificate that the historical evidence was not a fluke. The authors stress that promotion should trigger a forward-looking evaluation rather than immediate large-scale trading.
Key Results
The results highlight the massive inflation of error in standard practices:
- Naive Testing Fails: In the null experiments, simply running repeated unadjusted tests resulted in a 94.5% Familywise Error Rate (FWER). In other words, the system almost always promoted a "fake" strategy.
- PROMOTE Stays Safe: Under the same conditions, PROMOTE recorded a 0.0% FWER in the fixed-family audit and 1.77% in the adaptive-birth audit, staying well within the 5% safety limit.
- Power and Delay: When a real 15-basis-point signal was injected, PROMOTE achieved 70.1% target power. However, it required a median of 2,327 sessions to promote, whereas a naive (but unsafe) test might trigger in just 125 sessions.
In the real-world market replay, PROMOTE promoted zero strategies. While this might seem disappointing, the authors point out that a naive inspection would have promoted a "January effect" rule that actually lost money in the subsequent years. PROMOTE's "silence" was the correct statistical response to a weak library.
Limitations and Future Work
- Historical, Not Predictive: The authors are clear: this protocol controls historical false promotion. It does not guarantee that a strategy's edge will persist in the future, especially if the market regime changes.
- Clipping: To ensure statistical safety, returns are "clipped." While this prevents outliers from breaking the math, it could also suppress information about rare, massive economic losses.
- Retrospective Design: The library of 24 rules used in the market replay was designed retrospectively. A true test of the system requires a "live" research agent operating in real-time.
- Future Directions: The authors suggest that "expiring evidence" (where promotion certificates have a shelf life) and "funded revalidation" (re-testing a strategy for a new epoch) are necessary next steps for a production-ready system.
Summary updated