ABIDES-MARL: A Multi-Agent Reinforcement Learning Environment for Endogenous Price Formation and Execution in a Limit Order Book

Patrick Cheridito, Jean-Loup Dupret, Zhigang WuarXiv 2025 · DOI · arXiv

Abstract

We present ABIDES-MARL, a framework that combines a new multi-agent reinforcement learning (MARL) methodology with a new realistic limit-order-book (LOB) simulation system to study equilibrium behavior in complex financial market games. The system extends ABIDES-Gym by decoupling state collection from kernel interruption, enabling synchronized learning and decision-making for multiple adaptive agents while maintaining compatibility with standard RL libraries. It preserves key market features such as price-time priority and discrete tick sizes. Methodologically, we use MARL to approximate equilibrium-like behavior in multi-period trading games with a finite number of heterogeneous agents—an informed trader, a liquidity trader, noise traders, and competing market makers—all with individual price impacts. This setting bridges optimal execution and market microstructure by embedding the liquidity trader's optimization problem within a strategic trading environment. We validate the approach by solving an extended Kyle model within the simulation system, recovering the gradual price discovery phenomenon. We then extend the analysis to a liquidity trader's problem where market liquidity arises endogenously and show that, at equilibrium, execution strategies shape market-maker behavior and price dynamics. ABIDES-MARL provides a reproducible foundation for analyzing equilibrium and strategic adaptation in realistic markets and contributes toward building economically interpretable agentic AI systems for finance.

Introduction

In traditional financial modeling, markets are often treated as exogenous forces—static price processes that react to a trader's actions through fixed, formulaic impact functions. However, real-world markets are fundamentally endogenous: large trades signal information to other participants, who then adjust their quotes and expectations accordingly. While modern "optimal execution" research has made strides, agents are typically trained against historical data or simplified stochastic processes, failing to capture the strategic feedback loops inherent in multi-agent environments.

ABIDES-MARL addresses this gap by bridging theoretical market microstructure and practical trading. By extending the high-fidelity ABIDES simulator, the framework allows for the study of how market phenomena, such as price discovery and liquidity, emerge naturally from the strategic interactions of heterogeneous agents.

The ABIDES-MARL Framework

The core technical innovation of ABIDES-MARL is the introduction of a synchronized kernel architecture. Previous iterations, such as ABIDES-Gym, were limited to single-agent reinforcement learning because they lacked a mechanism to pause the simulation for multiple agents simultaneously without disrupting the underlying discrete-event logic.

Figure 1: The ABIDES-MARL architecture. The system decouples state collection from kernel interruption using a StopSignalAgent, enabling synchronized multi-agent interactions.

As shown in Figure 1, the framework utilizes a StopSignalAgent to coordinate all RL participants. When the ABIDES Kernel pops a WakeupMsg from the message queue, the StopSignalAgent halts the simulation and broadcasts the raw states of all RL agents to the Gym Environment. This environment processes standardized observations and rewards, allowing policy networks to generate actions. These actions are then transformed into ABIDES-compatible formats and applied, after which the simulation resumes. This protocol ensures that all agents receive updates and submit actions in a synchronized, controlled manner, maintaining compatibility with standard RL libraries like Ray RLLib.

Agent Archetypes and Market Dynamics

The environment supports four distinct agent archetypes to simulate a realistic market ecosystem:

  1. Informed Trader: Possesses knowledge of the "true" fundamental value of an asset and seeks to profit from this information advantage.
  2. Liquidity Trader: Tasked with an "acquisition problem," needing to buy or sell a target volume by a specific deadline.
  3. Market Makers: Compete to provide liquidity by posting limit orders, attempting to infer the true asset value from observed order flow.
  4. Noise Traders: Provide stochastic background activity to simulate general market participation.

Agents are trained using Proximal Policy Optimization (PPO). The framework is flexible enough to support both linear policies, which align with classical economic models, and nonlinear policies utilizing deep neural networks.

Experimental Results

Validation of the Kyle Model

The system was validated by replicating the seminal Kyle (1985) model. In this setup, the simulation successfully recovered the gradual price discovery phenomenon, where prices converge toward the fundamental value over time. The researchers were able to empirically measure the "half-life" of information decay, demonstrating that the MARL approach can approximate theoretical equilibrium behavior in a high-fidelity LOB environment.

Execution Strategy Performance

The authors compared various execution strategies, including Time-Weighted Average Price (TWAP), Volume-Weighted Average Price (VWAP), and PPO-based agents. In linear settings, TWAP proved remarkably robust. However, in nonlinear settings, a "PPO-Single" agent—one retrained against fixed, previously learned opponents—achieved the lowest implementation shortfall. This suggests that while multi-agent PPO is powerful, it can suffer from adversarial "arms races" during simultaneous training, where agents over-optimize against each other's evolving strategies.

Key Insights and Discussion

A primary finding of this research is that liquidity is a reaction rather than a constant. In ABIDES-MARL, liquidity arises endogenously; if a liquidity trader acts predictably, market makers learn to adjust their quotes aggressively, increasing the trader's execution costs.

Furthermore, the study highlights the risks of unconstrained nonlinearity. When using deep neural networks, informed traders can learn to manipulate market makers' beliefs, driving prices to artificial levels to execute trades more favorably. The researchers found that maintaining a strategic balance is crucial; a market consisting only of liquidity traders and market makers tends to become degenerate, with market makers dominating. Reintroducing the informed trader acts as a necessary counterweight, restoring meaningful price discovery.

Conclusion and Future Work

ABIDES-MARL provides a robust, reproducible foundation for analyzing strategic adaptation in realistic financial markets. While the current model captures essential dynamics, future work aims to increase agent diversity and capture more complex "stylized facts" of financial markets, such as volatility clustering. Additionally, the authors propose integrating Large Language Models (LLMs) as "macro analysts" that provide textual signals to MARL agents, moving toward more sophisticated, economically interpretable agentic AI systems.

Summary updated

Abstract

We present ABIDES-MARL, a framework that combines a new multi-agent reinforcement learning (MARL) methodology with a new realistic limit-order-book (LOB) simulation system to study equilibrium behavior in complex financial market games. The system extends ABIDES-Gym by decoupling state collection from kernel interruption, enabling synchronized learning and decision-making for multiple adaptive agents while maintaining compatibility with standard RL libraries. It preserves key market features such as price-time priority and discrete tick sizes. Methodologically, we use MARL to approximate equilibrium-like behavior in multi-period trading games with a finite number of heterogeneous agents-an informed trader, a liquidity trader, noise traders, and competing market makers-all with individual price impacts. This setting bridges optimal execution and market microstructure by embedding the liquidity trader's optimization problem within a strategic trading environment. We validate the approach by solving an extended Kyle model within the simulation system, recovering the gradual price discovery phenomenon. We then extend the analysis to a liquidity trader's problem where market liquidity arises endogenously and show that, at equilibrium, execution strategies shape market-maker behavior and price dynamics. ABIDES-MARL provides a reproducible foundation for analyzing equilibrium and strategic adaptation in realistic markets and contributes toward building economically interpretable agentic AI systems for finance.
BibTeX
@misc{cheridito2025abides,
  author = {Patrick Cheridito and Jean-Loup Dupret and Zhigang Wu},
  title = {{ABIDES-MARL: A Multi-Agent Reinforcement Learning Environment for Endogenous Price Formation and Execution in a Limit Order Book}},
  year = {2025},
  eprint = {2511.02016},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.02016},
  howpublished = {arXiv},
}

Loading related content...

Figure