Towards Inverse Reinforcement Learning for Limit Order Book Dynamics

Jacobo Roa-Vicens, Cyrine Chtourou, Angelos Filos, Francisco Rullan, Yarin Gal, Ricardo SilvaarXiv 2019 · OpenReview

Abstract

Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquiring such reward functions through inference, allowing to generalize the resulting policy to states not observed in the past. This paper investigates whether IRL can infer such rewards from agents within real financial stochastic environments: limit order books (LOB). We introduce a simple one-level LOB, where the interactions of a number of stochastic agents and an expert trading agent are modelled as a Markov decision process. We consider two cases for the expert's reward: either a simple linear function of state features; or a complex, more realistic non-linear function. Given the expert agent's demonstrations, we attempt to discover their strategy by modelling their latent reward function using linear and Gaussian process (GP) regressors from previous literature, and our own approach through Bayesian neural networks (BNN). While the three methods can learn the linear case, only the GP-based and our proposed BNN methods are able to discover the non-linear reward case. Our BNN IRL algorithm outperforms the other two approaches as the number of samples increases. These results illustrate that complex behaviours, induced by non-linear reward functions amid agent-based stochastic scenarios, can be deduced through inference, encouraging the use of inverse reinforcement learning for opponent-modelling in multi-agent systems.

1. Introduction

In high-frequency trading, market participants can observe the actions of others—such as buy and sell orders in a Limit Order Book (LOB)—but the underlying motivations driving these actions remain hidden. Understanding these latent "reward functions" is critical for constructing realistic market simulations and predicting competitor behavior. However, financial environments are notoriously challenging due to their high stochasticity, non-stationarity, and complex multi-agent dynamics.

Traditional Inverse Reinforcement Learning (IRL) often assumes that an agent's reward is a simple linear combination of market features. In practice, trading strategies are frequently non-linear, shaped by complex factors like risk aversion. This paper addresses the challenge of inferring these non-linear rewards in a noisy, multi-agent financial setting by introducing a novel Bayesian Neural Network (BNN) approach to IRL.

2. Methodology

The authors model a simplified "one-level" LOB as a Markov Decision Process (MDP). In this environment, an "expert" agent interacts with several stochastic trading agents. The objective is to recover the expert's reward function based solely on observed trajectories.

The study compares three distinct IRL frameworks:

  1. Maximum Entropy (MaxEnt) IRL: A standard baseline assuming a linear mapping of state features to rewards.
  2. Gaussian Process IRL (GPIRL): A flexible approach using Gaussian Processes to model non-linear rewards.
  3. BNN IRL (Proposed): A two-step process where point estimates of rewards are first derived via the MaxEnt objective, followed by training a Bayesian Neural Network to learn the mapping between market states and these inferred rewards.
The Inverse Reinforcement Learning (IRL) framework loop, illustrating the interaction between the RL solver and the reward optimization process.

By utilizing a BNN, the model can approximate universal functions to handle non-linearity while simultaneously accounting for uncertainty in the demonstration data.

3. Experimental Setup

Experiments were conducted using synthetic data from a custom LOB simulator featuring $N=3$ trading agents with varying levels of randomness and one expert agent. The models were tested on datasets ranging from 512 to 16,384 expert trajectories to evaluate performance scaling. Two ground-truth reward scenarios were established: a simple linear hit count reward and a complex exponential reward representing risk-averse behavior.

4. Results and Discussion

Performance was quantified using Expected Value Difference (EVD), which measures the gap between the reward earned by the true optimal policy and the policy derived from inferred rewards.

While all three methods successfully recovered the linear reward function, the linear MaxEnt model failed in the non-linear case. Both GPIRL and the proposed BNN IRL were successful in the non-linear scenario, but BNN IRL demonstrated superior performance as the volume of demonstration data increased.

Performance comparison (Expected Value Difference) for the exponential reward case across different numbers of demonstrations.

As shown in the performance analysis above, the BNN approach (yellow squares) significantly outperforms GPIRL (red diamonds) as the number of demonstrations grows. Beyond accuracy, the BNN approach offers better computational scalability ($O(n)$ complexity) compared to the $O(n^3)$ complexity of Gaussian Processes, making it a more viable candidate for high-dimensional, real-world LOB applications.

5. Conclusion and Future Work

This study demonstrates that complex trading behaviors induced by non-linear rewards can be effectively inferred in stochastic agent-based environments. The proposed BNN IRL framework provides a scalable and accurate path for opponent modeling in multi-agent systems.

Future research will focus on scaling this approach to "deep" multi-level LOBs and testing the framework on real-world historical exchange data to evaluate its robustness against market regime shifts and extreme volatility.

Summary updated

Abstract

Multi-agent learning is a promising method to simulate aggregate competitive behaviour in finance. Learning expert agents' reward functions through their external demonstrations is hence particularly relevant for subsequent design of realistic agent-based simulations. Inverse Reinforcement Learning (IRL) aims at acquiring such reward functions through inference, allowing to generalize the resulting policy to states not observed in the past. This paper investigates whether IRL can infer such rewards from agents within real financial stochastic environments: limit order books (LOB). We introduce a simple one-level LOB, where the interactions of a number of stochastic agents and an expert trading agent are modelled as a Markov decision process. We consider two cases for the expert's reward: either a simple linear function of state features; or a complex, more realistic non-linear function. Given the expert agent's demonstrations, we attempt to discover their strategy by modelling their latent reward function using linear and Gaussian process (GP) regressors from previous literature, and our own approach through Bayesian neural networks (BNN). While the three methods can learn the linear case, only the GP-based and our proposed BNN methods are able to discover the non-linear reward case. Our BNN IRL algorithm outperforms the other two approaches as the number of samples increases. These results illustrate that complex behaviours, induced by non-linear reward functions amid agent-based stochastic scenarios, can be deduced through inference, encouraging the use of inverse reinforcement learning for opponent-modelling in multi-agent systems.
BibTeX
@misc{roavicens2019towards,
  author = {Jacobo Roa-Vicens and Cyrine Chtourou and Angelos Filos and Francisco Rullan and Yarin Gal and Ricardo Silva},
  title = {{Towards Inverse Reinforcement Learning for Limit Order Book Dynamics}},
  year = {2019},
  eprint = {1906.04813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.04813},
}
Figure