Stock price formation: useful insights from a multi-agent reinforcement learning model

Johann Lussange, Sacha Bourgeois‐Gironde, Stefano Palminteri, Boris GutkinarXiv 2019 · DOI · arXiv

Abstract

In the past, financial stock markets have been studied with previous generations of multi-agent systems (MAS) that relied on zero-intelligence agents, and often the necessity to implement so-called noise traders to sub-optimally emulate price formation processes. However recent advances in the fields of neuroscience and machine learning have overall brought the possibility for new tools to the bottom-up statistical inference of complex systems. Most importantly, such tools allows for studying new fields, such as agent learning, which in finance is central to information and stock price estimation. We present here the results of a new generation MAS stock market simulator, where each agent autonomously learns to do price forecasting and stock trading via model-free reinforcement learning, and where the collective behaviour of all agents decisions to trade feed a centralised double-auction limit order book, emulating price and volume microstructures. We study here what such agents learn in detail, and how heterogenous are the policies they develop over time. We also show how the agents learning rates, and their propensity to be chartist or fundamentalist impacts the overall market stability and agent individual performance. We conclude with a study on the impact of agent information via random trading.

Introduction

Traditional financial models often rely on "zero-intelligence" agents—entities that follow fixed, hard-coded rules—or "noise traders" implemented as a heuristic to emulate price movement. These models fail to capture the epistemological reality of markets: that prices are formed by agents who learn, adapt, and react to one another. This research moves financial modeling away from these simplistic agents toward a more sophisticated, biologically-inspired approach. By applying reinforcement learning (RL) to a multi-agent system (MAS), we explore how individual learning dynamics aggregate into macro-level market phenomena like "stylized facts," crashes, and stability.

Methodology

The researchers developed a MAS stock market simulator centered around a centralized double-auction limit order book (LOB). The core innovation is a dual-algorithm architecture for each agent:

  1. Forecasting Algorithm ($F^i$): An RL agent that monitors price volatility and the fundamental value gap to predict future prices using various econometric tools (mean-reverting, averaging, or trend-following).
  2. Trading Algorithm ($T^i$): An RL agent that takes the forecast as input to decide on buy/sell/hold actions based on portfolio state and risk.

Agents are initialized with a reflexivity parameter ($\rho^i$), determining their bias toward fundamentalist or chartist strategies. The model was calibrated using London Stock Exchange (LSE) daily quotes from 2008 to 2018 to ensure the emergence of realistic "stylized facts" such as non-Gaussian returns and volatility clustering.

Results and Discussion

Policy Heterogeneity and the "Fundamentalist Advantage"

Analysis of agent performance reveals a stark contrast in strategy evolution. The most successful agents developed highly diverse and unique strategies, whereas the worst-performing agents' policies converged. This suggests that while there are many paths to profit, the paths to failure are remarkably similar.

Figure 2: Evolution of policy distance over time, showing higher heterogeneity among top-performing agents compared to the convergence of poor-performing agents.

Furthermore, the best-performing 10% of agents evolved to be primarily "fundamentalists" (low reflexivity), while the bottom 10% remained "chartists" (high reflexivity). In a learning-capable system, trend-following appears to be a sub-optimal strategy leading to poor individual performance.

Figure 1: Distribution of reflexivity parameters for the best and worst performing agents, highlighting the fundamentalist bias of successful agents.

Market Stability and the Role of Noise

The study highlights several counter-intuitive findings regarding market stability:

  • The Danger of High Learning Rates: Increasing the agent learning rate ($\zeta$) significantly increased the frequency of "tail events" or market crashes, suggesting that faster adaptation can lead to systemic fragility.
  • The "Noise" Stabilizer: Increasing the percentage of "noise traders" (random actors) actually decreased market crashes and bankruptcy rates. Randomness provides the liquidity and "friction" necessary to prevent the feedback loops that cause crashes.
  • Herding Impacts: Emulating the "best" agents (herding) is detrimental to stability. For herding levels above 50%, price volatility and crashes skyrocketed.

Conclusion

This work demonstrates that agent learning and heterogeneity are critical to understanding stock price formation. The findings suggest that market stability is improved by random actors, while herding behind top performers can trigger volatility. Future research will incorporate short selling, multi-asset portfolio diversification, and the impact of exogenous information asymmetry to further refine the model's realism.

Summary updated

Abstract

In the past, financial stock markets have been studied with previous generations of multi-agent systems (MAS) that relied on zero-intelligence agents, and often the necessity to implement so-called noise traders to sub-optimally emulate price formation processes. However recent advances in the fields of neuroscience and machine learning have overall brought the possibility for new tools to the bottom-up statistical inference of complex systems. Most importantly, such tools allows for studying new fields, such as agent learning, which in finance is central to information and stock price estimation. We present here the results of a new generation MAS stock market simulator, where each agent autonomously learns to do price forecasting and stock trading via model-free reinforcement learning, and where the collective behaviour of all agents decisions to trade feed a centralised double-auction limit order book, emulating price and volume microstructures. We study here what such agents learn in detail, and how heterogenous are the policies they develop over time. We also show how the agents learning rates, and their propensity to be chartist or fundamentalist impacts the overall market stability and agent individual performance. We conclude with a study on the impact of agent information via random trading.
BibTeX
@misc{lussange2019stock,
  author = {Johann Lussange and Sacha Bourgeois‐Gironde and Stefano Palminteri and Boris Gutkin},
  title = {{Stock price formation: useful insights from a multi-agent reinforcement learning model}},
  year = {2019},
  eprint = {1910.05137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.05137},
}
Figure