Abstract
Quantitative finance has had a long tradition of a bottom-up approach to complex systems inference via multi-agent systems (MAS). These statistical tools are based on modelling agents trading via a centralised order book, in order to emulate complex and diverse market phenomena. These past financial models have all relied on so-called zero-intelligence agents, so that the crucial issues of agent information and learning, central to price formation and hence to all market activity, could not be properly assessed. In order to address this, we designed a next-generation MAS stock market simulator, in which each agent learns to trade autonomously via model-free reinforcement learning. We calibrate the model to real market data from the London Stock Exchange over the years 2007 to 2018, and show that it can faithfully reproduce key market microstructure metrics, such as various price autocorrelation scalars over multiple time intervals. Agent learning thus enables model emulation of the microstructure with greater realism.
Introduction
Traditional bottom-up models in quantitative finance, known as Multi-Agent Systems (MAS), have historically relied on "zero-intelligence" agents—entities that trade based on simple, often random, rules. While these models can emulate some market phenomena, they fail to capture the crucial roles of information processing and learning in price formation. To truly understand market microstructure—the internal mechanics of how trades occur and prices change—we require agents that can adapt their strategies based on experience, much like real-world investors. This paper presents a simulator where autonomous agents learn to trade using model-free reinforcement learning, bridging the gap between simple rule-based models and the complex reality of financial markets.
Methodology
The simulator's core innovation is a dual-reinforcement learning architecture for every agent, interacting via a Double Auction Order Book. Each agent employs two distinct learning layers:
- Algorithm $\mathcal{F}$ (Forecasting): Learns to predict future prices by choosing between econometric tools (e.g., mean-reverting, moving-average, or trend-following) and weighing technical analysis against fundamental value.
- Algorithm $\mathcal{T}$ (Trading): Decides the specifics of the trade (buy, sell, or hold) and the aggressiveness of the limit price based on the forecast from Algorithm $\mathcal{F}$.
- Timing Filter ($G^i$): A specialized function that optimizes order timing to prevent system noise.
Data and Calibration
The model was calibrated using daily close prices and volumes for 640 continuously traded stocks from the London Stock Exchange (LSE) between 2007 and 2018. To provide a "ground truth" for fundamentalist trading strategies, the authors generated synthetic fundamental value time series as jump processes, modeled after real-world enterprise value metrics.
Results and Discussion
The interaction of learning agents leads to the emergence of complex market behaviors, such as volatility clustering, without explicit hard-coding. Agents autonomously learn to balance "chartist" (trend-following) and "fundamentalist" (value-based) strategies, mirroring the diversity found in actual markets.
The simulator's output for logarithmic price returns and volatilities showed a satisfactory topological fit compared to real LSE data across multiple time scales. Furthermore, the model successfully emulated the decaying autocorrelation of price returns, indicating that the learning process effectively eliminates obvious arbitrage opportunities. While a Random Forest classifier could still distinguish simulated from real data, non-linear mapping (UMAP) showed significant overlap, suggesting the simulator captures the core essence of market dynamics.
Conclusion
By moving away from zero-intelligence agents toward autonomous, learning entities, this research demonstrates that agent learning is essential for realistic financial simulations. Future work will focus on extending the model to the continuous domain using Deep Q-Learning or Policy Gradients and expanding to multivariate features to study covariance structures in multi-agent environments.
Summary updated