Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents

Eric Benhamou, Jean-Jacques Ohana, Beatrice Guez, Ethan Setrouk, Chamyl Saadi, Thomas JacquotarXiv 2026 · arXiv

TL;DR

The paper introduces PROMOTE, a rigorous protocol for financial research that prevents "backtest overfitting" by treating strategy discovery as a budget-constrained betting game. By using anytime-valid e-processes, it allows researchers and AI agents to continuously monitor and add new trading rules without inflating false-positive rates, ensuring that "promoted" strategies have genuine statistical evidence rather than just being lucky.

What Problem Does This Paper Address?

In quantitative finance, the "backtest" is the standard way to vet a trading strategy. However, traditional backtesting is plagued by data snooping and optional stopping. A researcher (or an autonomous AI agent) rarely runs just one test; they tweak parameters, change the asset universe, or drop unpromising rules based on interim results.

Treating the final "winning" rule as if it came from a single, fixed test is a statistical fallacy. At machine speeds, AI agents can generate thousands of variations, making it almost certain that some will look profitable purely by chance. This leads to "overfitted" strategies that perform brilliantly in the past but fail immediately upon deployment. The authors seek a way to authorize continuous, adaptive research while maintaining a strict guarantee on the probability of promoting a "dud" strategy.

Methods

The authors propose a protocol called PROMOTE, built on the foundations of anytime-valid inference and e-processes.

  1. Statistical Wealth Budget: The research program starts with a total "wealth" of 1. Every time a new candidate strategy $h$ is created, it is assigned a portion of this wealth ($\pi_h$), such that the sum of all assigned wealth never exceeds 1.
  2. Predictable Birth: A strategy must be "born" at a specific time. It cannot "inherit" evidence from before its birth; it only starts accumulating evidence from returns realized after it is registered.
  3. E-Processes as Evidence: For each strategy, the protocol tracks an e-process—a non-negative supermartingale that represents the "winnings" from betting against the null hypothesis (that the strategy has no edge).
  4. The Promotion Threshold: A strategy is promoted only if its e-process crosses a boundary of $1/(\alpha \pi_h)$, where $\alpha$ is the desired false-promotion rate (e.g., 5%).
  5. Cost-Awareness: The protocol explicitly includes transaction costs and uses "clipped" returns to ensure the test remains valid even if the underlying return distribution has heavy tails.

This approach turns the research process into an auditable ledger. You can add as many strategies as you want, and you can stop whenever you see a crossing, provided you have the "statistical budget" to fund them.

Datasets

The authors evaluate PROMOTE using three distinct setups:

  • Shared-Sign Null Paths: 3,000 Monte Carlo simulations using randomized signs of real historical returns to create a "null" environment where no strategy has a true edge.
  • Exact Mean-Zero Experiment: 3,000 simulated research programs where an agent selects candidate identities and birth times based on past performance, testing the protocol's resilience to adaptive selection.
  • Public Market Replay: A "pseudo-out-of-sample" replay using real-world data from March 2006 to May 2026 (including projected dates). This includes 24 hypotheses across equity ETFs (SPY, QQQ, etc.), defensive ETFs (IEF, GLD), and Fama-French industry portfolios.

Key Insights and Ideas

  • Research as a Ledger: The most important takeaway is that research is a process, not a single event. PROMOTE formalizes this by requiring an append-only ledger of every idea, including the ones that failed.
  • The Price of Validity: Validity isn't free. The protocol shows that to be "anytime-valid," you must wait longer for evidence to accumulate. A "lucky" naive test might trigger early, but PROMOTE requires persistent evidence to cross its higher, wealth-adjusted boundary.
  • Independence is Not Required: Unlike many multi-comparison corrections (like Bonferroni), this method handles dependent candidates (e.g., two strategies that trade similar assets) without losing its safety guarantee.
  • Separation of Discovery and Deployment: A "promotion" is not a guarantee of future profit; it is a certificate that the historical evidence was not a fluke. The authors stress that promotion should trigger a forward-looking evaluation rather than immediate large-scale trading.

Key Results

The results highlight the massive inflation of error in standard practices:

  • Naive Testing Fails: In the null experiments, simply running repeated unadjusted tests resulted in a 94.5% Familywise Error Rate (FWER). In other words, the system almost always promoted a "fake" strategy.
  • PROMOTE Stays Safe: Under the same conditions, PROMOTE recorded a 0.0% FWER in the fixed-family audit and 1.77% in the adaptive-birth audit, staying well within the 5% safety limit.
  • Power and Delay: When a real 15-basis-point signal was injected, PROMOTE achieved 70.1% target power. However, it required a median of 2,327 sessions to promote, whereas a naive (but unsafe) test might trigger in just 125 sessions.
Figure 2: Fixed-family diagnostics. Left: familywise crossings over candidates and inspection times; the inset makes the 0.7% per-stream result and zero-count Wilson intervals visible. Right: target promotion probability under an aligned shift to each ETF component return; shaded regions are 95% Wilson intervals. The repeated-test curve is diagnostic, not a matched-error comparator.

In the real-world market replay, PROMOTE promoted zero strategies. While this might seem disappointing, the authors point out that a naive inspection would have promoted a "January effect" rule that actually lost money in the subsequent years. PROMOTE's "silence" was the correct statistical response to a weak library.

Figure 3: Pseudo-out-of-sample evidence and economic ledgers, January 2015–May 2026. Left: the six largest mixture e-processes; the artifact retains all 24 paths. Right: portfolio wealth after exact five-basis-point one-way turnover costs. The funded gate stays at one because no rule crosses; cash is assigned zero yield in this diagnostic ledger.

Limitations and Future Work

  • Historical, Not Predictive: The authors are clear: this protocol controls historical false promotion. It does not guarantee that a strategy's edge will persist in the future, especially if the market regime changes.
  • Clipping: To ensure statistical safety, returns are "clipped." While this prevents outliers from breaking the math, it could also suppress information about rare, massive economic losses.
  • Retrospective Design: The library of 24 rules used in the market replay was designed retrospectively. A true test of the system requires a "live" research agent operating in real-time.
  • Future Directions: The authors suggest that "expiring evidence" (where promotion certificates have a shelf life) and "funded revalidation" (re-testing a strategy for a new epoch) are necessary next steps for a production-ready system.

Summary updated

Abstract

Financial research agents can generate and monitor trading rules faster than a human can document the resulting selection process. Candidates appear at data-dependent times, weak rules are abandoned, and promising ones are watched until they cross a reporting threshold. We introduce PROMOTE, a protocol that assigns every candidate predictable testing wealth at birth and accumulates a cost-aware e-process only from subsequent returns. If lifetime allocations sum to at most one, a weighted Ville argument bounds the probability of ever promoting a rule whose declared bounded net score has always had nonpositive conditional mean. The result permits dependent candidates, adaptive births, and indefinite monitoring. We audit the implementation in two settings. A retrospective pseudo-out-of-sample replay contains 24 panel--rule hypotheses, 3,000 shared-sign null paths, injected alternatives, and public ETF and industry data. Repeated unadjusted tests produce a family crossing in 94.5% of null paths, versus 0.0% for PROMOTE; at a 15-bp aligned ETF-component perturbation, PROMOTE has 70.1% target power and median crossing session 2,327. A separate exact mean-zero experiment lets an agent select candidate identities and birth times from past scores; the funded gate has 1.77% false-family crossings across 3,000 programs. The market replay produces zero promotions after costs. The certificate controls historical false promotion under the declared null; it is not a certificate that an earlier edge remains profitable at deployment. AI-use disclosure. Generative AI tools, including OpenAI Codex, assisted manuscript preparation. For this preprint version, Codex assisted with LaTeX editing, author and venue metadata, compilation, and layout checks.
BibTeX
@misc{benhamou2026can,
  author = {Eric Benhamou and Jean-Jacques Ohana and Beatrice Guez and Ethan Setrouk and Chamyl Saadi and Thomas Jacquot},
  title = {{Can We Stop the Backtest Yet? Anytime-Valid Promotion Tests for Adaptive Financial Research Agents}},
  year = {2026},
  eprint = {2611.03384},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03384},
}
Figure