Adversarial recovery of agent rewards from latent spaces of the limit order book

Jacobo Roa-Vicens, Yuanbo Wang, Virgile Mison, Yarin Gal, Ricardo Silva

TL;DR

This paper demonstrates a robust way to "reverse-engineer" the strategies of expert trading agents by combining Adversarial Inverse Reinforcement Learning (AIRL) with compressed "World Models" of the stock market. By training in a latent space representation of the limit order book (LOB), the authors show that AIRL can recover reward functions that remain effective even when market dynamics shift, significantly outperforming standard imitation learning techniques.

What Problem Does This Paper Address?

Reinforcement Learning (RL) has seen massive success in games, but applying it to high-stakes financial markets is a nightmare. You can't simply let an agent learn by "trial and error" when every error costs real capital. Furthermore, manually designing a perfect reward function—the mathematical signal that tells an agent what to prioritize—is incredibly difficult in a complex, non-stationary environment like a stock exchange.

Inverse Reinforcement Learning (IRL) offers a solution: instead of telling the agent what to do, we observe an expert and infer their underlying reward function. However, IRL is "ill-posed"—many different rewards could explain the same behavior. Moreover, if the market's behavior (dynamics) changes, a reward function learned via simple imitation often fails. This paper tackles the challenge of recovering rewards that are robust to these changing market regimes.

Methods

The authors employ a two-stage pipeline to make the problem tractable:

  1. World Models in Latent Space: Raw Limit Order Book (LOB) data is high-dimensional and noisy. The authors use a CNN-based autoencoder to compress this data into a 12-dimensional latent vector ($z$). They then use a Recurrent Neural Network (RNN) combined with a Mixture Density Network (MDN) to predict future states. This "World Model" acts as a simulator where agents can be trained safely.

    Figure 2: Model for prediction of latent space transitions. Our RL processes are run in the latent state space.
  2. Adversarial Learning: They compare two adversarial approaches:

  3. GAIL (Generative Adversarial Imitation Learning): Directly learns a policy to mimic the expert.
  4. AIRL (Adversarial Inverse Reinforcement Learning): A more advanced approach that attempts to "disentangle" the agent's reward from the environment's dynamics.

By training AIRL within the latent space of the World Model, the authors aim to recover a reward function that represents the expert's intent, making it transferable to new market conditions.

Datasets

The study uses real-world tick data from Tencent Holdings Ltd. (HK-700) traded on the Hong Kong stock exchange.

  • Training Period: 60 trading days (January to March 2018).
  • Testing Period: 20 trading days (April 2018).
  • State Representation: Each state consists of 10 consecutive LOB data ticks across 3 levels of the book, capturing features like bid/ask prices and volumes.

Key Insights and Ideas

  • The Power of Compression: Training RL agents on raw LOB data is computationally expensive. By moving to a 12D latent space, the authors created a "distilled" version of the market that retains essential transition dynamics while being much easier for algorithms to digest.
  • Disentanglement Matters: The "aha moment" is that AIRL outperforms GAIL specifically because it recovers a reward function, not just a policy. In finance, the "rules" of the market (dynamics) are non-stationary. AIRL’s ability to separate the agent's goals from the environment's current state allows it to adapt better when volatility or price trends shift.
  • Latent Space Transferability: The results confirm that policies and rewards learned in a latent-space simulation can be successfully transferred back to "real" data regimes (the April test set) and remain profitable.

Key Results

The experiments used three types of experts: Advantage Actor-Critic (A2C), Double DQN, and Policy Gradient (PG). In every single case, AIRL outperformed GAIL in terms of cumulative reward recovered.

Expert Type Expert Avg. Return AIRL Avg. Return GAIL Avg. Return
A2C 5047.3 2433.0 1563.1
DQN 4177.4 2151.1 1574.5
PG 1483.3 1267.7 1029.1

As shown in the training curves, AIRL's performance continues to climb as iterations increase, while GAIL often plateaus earlier, struggling to capture the nuances of the expert's strategy in a changing environment.

Comparison of rewards obtained through AIRL and GAIL across different expert types.

Limitations and Future Work

  • Performance Gap: While AIRL beats GAIL, it still only recovers about 50% of the total cumulative reward of the best experts (like A2C). There is still a significant "information loss" during the recovery process.
  • Expert Optimality: The model assumes the observed trajectories come from an optimal expert. In real-world applications, "expert" data might be noisy or sub-optimal, which could confuse the adversarial discriminator.
  • Future Directions: The authors suggest exploring more complex LOB features and testing the robustness of these recovered rewards across even longer time horizons and more diverse market regimes.

Summary updated

Abstract

Inverse reinforcement learning has proved its ability to explain state-action trajectories of expert agents by recovering their underlying reward functions in increasingly challenging environments. Recent advances in adversarial learning have allowed extending inverse RL to applications with non-stationary environment dynamics unknown to the agents, arbitrary structures of reward functions and improved handling of the ambiguities inherent to the ill-posed nature of inverse RL. This is particularly relevant in real time applications on stochastic environments involving risk, like volatile financial markets. Moreover, recent work on simulation of complex environments enable learning algorithms to engage with real market data through simulations of its latent space representations, avoiding a costly exploration of the original environment. In this paper, we explore whether adversarial inverse RL algorithms can be adapted and trained within such latent space simulations from real market data, while maintaining their ability to recover agent rewards robust to variations in the underlying dynamics, and transfer them to new regimes of the original environment.
BibTeX
@misc{roavicens2019adversarial,
  author = {Jacobo Roa-Vicens and Yuanbo Wang and Virgile Mison and Yarin Gal and Ricardo Silva},
  title = {{Adversarial recovery of agent rewards from latent spaces of the limit order book}},
  year = {2019},
  eprint = {1912.04242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.04242},
}
Figure