Modelling Stock Markets by Multi-agent Reinforcement Learning

Johann Lussange, Ivan Lazarevich, Sacha Bourgeois‐Gironde, Stefano Palminteri, Boris GutkinComputational 2020 · DOI

TL;DR

This paper introduces a novel multi-agent reinforcement learning (MARL) stock market simulator where each agent learns to trade autonomously. By incorporating realistic agent behaviors and learning mechanisms, the simulator can accurately reproduce key market microstructure phenomena, offering a more sophisticated tool for financial research than previous models.

What Problem Does This Paper Address?

Quantitative finance has long sought to understand complex market phenomena through bottom-up, multi-agent system (MAS) models. However, traditional MAS models often relied on "zero-intelligence" agents, limiting their ability to capture crucial aspects like agent information processing and learning, which are central to price formation. This gap prevents a deeper understanding of market dynamics and stability. This paper addresses this by developing a next-generation MAS stock market simulator where agents learn to trade using reinforcement learning, allowing for the assessment of agent learning and its emergent impact on market behavior.

Methods

The core of this work is a multi-agent system (MAS) stock market simulator. The key innovation lies in equipping each agent with two distinct reinforcement learning (RL) algorithms:

  • Forecasting Algorithm (F): This algorithm learns to forecast future stock prices. It considers states related to longer-term and shorter-term price volatility, as well as the gap between the agent's fundamental valuation and the current market price. The agent can choose forecasting tools (mean-reverting, moving-average, trend-following), the time interval for the forecast, and the weighting between its own technical forecast and fundamental valuation.
  • Trading Algorithm (T): This algorithm uses the price forecast from F to decide on trading actions. It considers states related to price trends, volatility, risk-free assets, stock holdings, and trading volume. The agent can choose to send a transaction order (buy, sell, or hold) and set the price based on supply/demand dynamics and its own transaction gesture.

The simulator models a centralized order book and incorporates realistic elements like broker fees, risk-free rates, and dividend yields. The agents' learning is driven by direct policy search, allowing them to explore and exploit actions to maximize their rewards.

Datasets

The MAS stock market simulator was calibrated and tested using real stock market data from the London Stock Exchange (LSE). Specifically, the dataset includes:

  • Daily closing prices and volumes for 4,313 stocks from January 15, 2007, to January 19, 2018.
  • Data was filtered to remove stock splits and include only continuously traded stocks, resulting in a universe of 640 stocks.

The paper does not introduce new benchmark datasets but uses this real-world data for calibration and validation.

Key Insights and Ideas

  • Agent Learning as an Emergent Property: The study demonstrates that realistic market microstructure phenomena, such as price autocorrelations and volatility clustering, can emerge from the collective learning behavior of autonomous agents, rather than being explicitly programmed.
  • Reinforcement Learning for Financial Modeling: RL proves to be a powerful framework for modeling complex agent decision-making in financial markets, allowing agents to adapt their strategies based on market feedback.
  • Fundamental vs. Chartist Behavior: Agents can learn to balance or prioritize between chartist (market price-driven) and fundamentalist (intrinsic value-driven) trading strategies, reflecting real-world investor behavior.
  • Reproducing Stylized Facts: The MARL simulator successfully emulates many of the "stylized facts" of financial markets, which are empirical regularities that challenge traditional economic models.

Key Results

  • Accurate Emulation of Market Statistics: The simulator shows a close qualitative agreement in shape with real stock market data for key indicators like logarithmic price returns, price volatilities at different time scales, and autocorrelations of price returns.
  • Classification Accuracy: Random forest classification models trained on features extracted from simulated and real time-series data achieved reasonable accuracy (around 62-64% for PCA/UMAP embeddings, and higher for specific feature sets), indicating that the simulated data captures essential characteristics of real market dynamics.
  • Model Sensitivity: The study found that the model is relatively linear with respect to the number of agents and gesture scalar, while being less affected by large drawdown thresholds.

Limitations and Future Work

The authors acknowledge several limitations:

  • Reliance on Virtual Fundamental Generation: The fundamental values of stocks are generated virtually, not based on real company data.
  • Absence of Portfolio Diversification: The model currently focuses on a single stock and does not account for diversification across multiple assets or asset classes.
  • Limited Trading Strategies: The agents are restricted to a long-only equity strategy, excluding short-selling, leveraging, derivatives, market orders, etc.
  • Absence of Intraday/Seasonal Effects: The model operates on daily data and does not capture intraday market activity or seasonal calendar effects.
  • No Legal/Regulatory Constraints: Real-world legal and regulatory frameworks are not included.

Future work suggested includes:

  • Multivariate Portfolio Management: Extending the model to study covariance structures across multiple stocks and portfolio risk management.
  • Continuous Domain Extension: Adapting the reinforcement learning framework to continuous action and state spaces (e.g., policy gradients, deep Q-learning).
  • Incorporating Psychological Traits: Utilizing the RL framework to implement psychological traits from decision theory and behavioral economics to study agent learning and cognition's influence on markets.
  • Deep Learning for Classification: Exploring deep neural networks (like 1D-CNNs) for more effective time-series classification and model tuning.

Summary updated

BibTeX
@inproceedings{lussange2020modelling,
  author = {Johann Lussange and Ivan Lazarevich and Sacha Bourgeois‐Gironde and Stefano Palminteri and Boris Gutkin},
  title = {{Modelling Stock Markets by Multi-agent Reinforcement Learning}},
  booktitle = {Computational},
  year = {2020},
  doi = {10.1007/s10614-020-10038-w},
}
Figure