Stock market microstructure inference via multi-agent reinforcement learning

Johann Lussange, Ivan Lazarevich, Sacha Bourgeois‐Gironde, Stefano Palminteri, Boris GutkinarXiv 2019 · DOI · arXiv

Abstract

Quantitative finance has had a long tradition of a bottom-up approach to complex systems inference via multi-agent systems (MAS). These statistical tools are based on modelling agents trading via a centralised order book, in order to emulate complex and diverse market phenomena. These past financial models have all relied on so-called zero-intelligence agents, so that the crucial issues of agent information and learning, central to price formation and hence to all market activity, could not be properly assessed. In order to address this, we designed a next-generation MAS stock market simulator, in which each agent learns to trade autonomously via model-free reinforcement learning. We calibrate the model to real market data from the London Stock Exchange over the years 2007 to 2018, and show that it can faithfully reproduce key market microstructure metrics, such as various price autocorrelation scalars over multiple time intervals. Agent learning thus enables model emulation of the microstructure with greater realism.

Introduction

Traditional bottom-up models in quantitative finance, known as Multi-Agent Systems (MAS), have historically relied on "zero-intelligence" agents—entities that trade based on simple, often random, rules. While these models can emulate some market phenomena, they fail to capture the crucial roles of information processing and learning in price formation. To truly understand market microstructure—the internal mechanics of how trades occur and prices change—we require agents that can adapt their strategies based on experience, much like real-world investors. This paper presents a simulator where autonomous agents learn to trade using model-free reinforcement learning, bridging the gap between simple rule-based models and the complex reality of financial markets.

Methodology

The simulator's core innovation is a dual-reinforcement learning architecture for every agent, interacting via a Double Auction Order Book. Each agent employs two distinct learning layers:

  1. Algorithm $\mathcal{F}$ (Forecasting): Learns to predict future prices by choosing between econometric tools (e.g., mean-reverting, moving-average, or trend-following) and weighing technical analysis against fundamental value.
  2. Algorithm $\mathcal{T}$ (Trading): Decides the specifics of the trade (buy, sell, or hold) and the aggressiveness of the limit price based on the forecast from Algorithm $\mathcal{F}$.
  3. Timing Filter ($G^i$): A specialized function that optimizes order timing to prevent system noise.

Data and Calibration

The model was calibrated using daily close prices and volumes for 640 continuously traded stocks from the London Stock Exchange (LSE) between 2007 and 2018. To provide a "ground truth" for fundamentalist trading strategies, the authors generated synthetic fundamental value time series as jump processes, modeled after real-world enterprise value metrics.

Synthetic fundamental price time series used as ground truth for agents.

Results and Discussion

The interaction of learning agents leads to the emergence of complex market behaviors, such as volatility clustering, without explicit hard-coding. Agents autonomously learn to balance "chartist" (trend-following) and "fundamentalist" (value-based) strategies, mirroring the diversity found in actual markets.

Comparison of true price signals against biased agent perceptions and tracking over time.

The simulator's output for logarithmic price returns and volatilities showed a satisfactory topological fit compared to real LSE data across multiple time scales. Furthermore, the model successfully emulated the decaying autocorrelation of price returns, indicating that the learning process effectively eliminates obvious arbitrage opportunities. While a Random Forest classifier could still distinguish simulated from real data, non-linear mapping (UMAP) showed significant overlap, suggesting the simulator captures the core essence of market dynamics.

Conclusion

By moving away from zero-intelligence agents toward autonomous, learning entities, this research demonstrates that agent learning is essential for realistic financial simulations. Future work will focus on extending the model to the continuous domain using Deep Q-Learning or Policy Gradients and expanding to multivariate features to study covariance structures in multi-agent environments.

Summary updated

Abstract

Quantitative finance has had a long tradition of a bottom-up approach to complex systems inference via multi-agent systems (MAS). These statistical tools are based on modelling agents trading via a centralised order book, in order to emulate complex and diverse market phenomena. These past financial models have all relied on so-called zero-intelligence agents, so that the crucial issues of agent information and learning, central to price formation and hence to all market activity, could not be properly assessed. In order to address this, we designed a next-generation MAS stock market simulator, in which each agent learns to trade autonomously via model-free reinforcement learning. We calibrate the model to real market data from the London Stock Exchange over the years $2007$ to $2018$, and show that it can faithfully reproduce key market microstructure metrics, such as various price autocorrelation scalars over multiple time intervals. Agent learning thus enables model emulation of the microstructure with greater realism.
BibTeX
@misc{lussange2019stock,
  author = {Johann Lussange and Ivan Lazarevich and Sacha Bourgeois‐Gironde and Stefano Palminteri and Boris Gutkin},
  title = {{Stock market microstructure inference via multi-agent reinforcement learning}},
  year = {2019},
  eprint = {1909.07748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.07748},
}
Figure