Evolving Quantitative Reasoning through Self-Play in Digital Twin Markets

Tianmi Ma, Wenxin Huang, Jiawei Du, Lin Li, Xian Zhong, Joey Tianyi ZhouICML 2026 · OpenReview

Large Language Models (LLMs) are brilliant strategists but notoriously unreliable calculators. In high-stakes domains like financial trading, a minor numerical hallucination or a misunderstood threshold can lead to catastrophic strategic risks. To bridge this gap, researchers have introduced DeMAgent, a novel framework that transforms LLMs from flawed calculators into high-level planners that dynamically write, test, and refine their own quantitative tools within a simulated "digital twin" market.

The Flaws in Current LLM Trading Agents

While LLMs excel at semantic reasoning—such as interpreting market news or formulating high-level strategies—they struggle with "symbolic" quantitative tasks. Current approaches attempt to fix this by equipping LLMs with a fixed "toolbox" of standard financial indicators (like RSI or MACD). However, these static tools fail to adapt to shifting market regimes.

Furthermore, evaluating trading agents using standard historical backtesting is fundamentally flawed. LLMs have often "memorized" historical price movements during their pre-training phase, leading to data leakage. Additionally, static backtesting cannot account for market impact—how an agent's own trades influence supply, demand, and subsequent prices.

A Decoupled Architecture: DeMAgent and DecoupledMarket

To solve these issues, the researchers propose a "decoupled" architecture that strictly separates strategic planning from numerical computation.

1. The DeMAgent Pipeline

Rather than forcing the LLM to compute numbers, DeMAgent assigns it to planning and analysis, delegating the math to executable code. The agent operates in a three-stage pipeline:

  • Market Analysis: The LLM distills complex, raw market data into structured, high-level abstractions.
  • Knowledge & Tool Creation (KTC): Instead of selecting from a pre-defined list of tools, the LLM writes Python code to create task-specific indicators (e.g., an "Adaptive Momentum-Volatility Ratio"). This code serves as a verifiable reasoning artifact; if the logic is flawed, the code fails safely rather than hallucinating a plausible number.
  • Evaluation & Refinement: The agent tests its newly minted tools in a simulator, receives performance-based feedback (evaluating stability and sensitivity), and iteratively refines its code.

2. DecoupledMarket: The Digital Twin Sandbox

To provide a rigorous testing ground, the authors built DecoupledMarket, a controllable "digital twin" market simulator. Unlike static historical backtesting, DecoupledMarket features endogenous price formation. Prices emerge in real-time based on the supply and demand generated by agents interacting within the simulation. This creates a completely "leakage-free" environment, forcing the agent to adapt to non-stationarity and the evolving behaviors of other market participants.

Key Results and Emergent Behaviors

The researchers evaluated DeMAgent across four distinct environments: the synthetic DecoupledMarket, NASDAQ (U.S. tech equities), CSI (Chinese market with unique constraints), and Cryptocurrency (high-volatility assets like BTC, SOL, and BNB).

  • Superior Performance: DeMAgent consistently outperformed classical technical strategies (like Buy & Hold or MACD) and standard Reinforcement Learning baselines (like PPO and TD3) across all tested markets.
  • Emergent Tool Creation: In a fascinating case study, the agent independently "invented" an adaptive momentum-volatility indicator (AMVR) to navigate a specific market regime. This demonstrates that LLMs can synthesize complex, novel financial concepts when placed in the right framework.
  • Model Scaling: The framework scales beautifully with more advanced foundation models. When powered by GPT-5-mini or Gemini-3-Pro, DeMAgent achieved significantly higher Total Returns (TR) and Sharpe Ratios (SR). In the digital twin market, a Gemini-3-Pro-powered DeMAgent reached a TR of 13.95% with zero maximum drawdown.
  • Real-World Generalization: Strategies and tools developed through self-play in the synthetic digital twin successfully transferred to real-world historical data, proving that the agent learns robust reasoning patterns rather than just overfitting to the simulation. Ablation studies confirmed that the dynamic Tool Creation module is the critical driver of this success.

Limitations and Future Outlook

While DeMAgent represents a significant leap forward, the authors note several limitations. The iterative self-play loop is computationally expensive, requiring significantly more API calls and introducing higher latency than standalone LLM inference. Furthermore, there is a substantial infrastructure gap between this high-level reasoning framework and the low-level, sub-second execution systems required for live trading.

Finally, the paper includes a crucial systemic risk warning: if highly adaptive, self-evolving agents were deployed at scale in real financial markets, they could inadvertently create feedback loops or "crowding" behaviors, potentially increasing market instability.

All code and data for this research are available at the authors' GitHub repository.

Summary updated

Abstract

Large language models (LLMs) demonstrate strong capabilities in high-level semantic reasoning and strategic planning, making them appealing for complex decision-making tasks; however, their quantitative reasoning remains unreliable despite recent progress in tool-augmented and structured inference. To address this limitation, we decouple reasoning from computation by assigning LLMs to planning, analysis, and result interpretation, while delegating numerical computation and statistical inference to specialized external tools. Rather than being hard-coded, these tools are constructed in a constrained and structured manner during planning as explicit intermediate reasoning artifacts, enabling adaptive and scenario-dependent quantitative reasoning. LLMs iteratively analyze tool outputs under diverse market conditions and leverage performance-based feedback to refine subsequent tool selection and construction, thereby forming a bounded self-evolving loop. We instantiate this process through self-play in a controllable digital twin market, DecoupledMarket, where LLM agents continuously test, compare, and adapt their strategies. By coupling high-level planning with robust quantitative execution, the proposed framework enhances the quantitative reliability of LLM-driven decision-making. All code and data are available at https://github.com/MTMQuantAI/Agent-Trading-Arena.git.
BibTeX
@inproceedings{ma2026evolving,
  author = {Tianmi Ma and Wenxin Huang and Jiawei Du and Lin Li and Xian Zhong and Joey Tianyi Zhou},
  title = {{Evolving Quantitative Reasoning through Self-Play in Digital Twin Markets}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=j4SxlIyvRg},
}
Figure