TL;DR
This paper demonstrates a robust way to "reverse-engineer" the strategies of expert trading agents by combining Adversarial Inverse Reinforcement Learning (AIRL) with compressed "World Models" of the stock market. By training in a latent space representation of the limit order book (LOB), the authors show that AIRL can recover reward functions that remain effective even when market dynamics shift, significantly outperforming standard imitation learning techniques.
What Problem Does This Paper Address?
Reinforcement Learning (RL) has seen massive success in games, but applying it to high-stakes financial markets is a nightmare. You can't simply let an agent learn by "trial and error" when every error costs real capital. Furthermore, manually designing a perfect reward function—the mathematical signal that tells an agent what to prioritize—is incredibly difficult in a complex, non-stationary environment like a stock exchange.
Inverse Reinforcement Learning (IRL) offers a solution: instead of telling the agent what to do, we observe an expert and infer their underlying reward function. However, IRL is "ill-posed"—many different rewards could explain the same behavior. Moreover, if the market's behavior (dynamics) changes, a reward function learned via simple imitation often fails. This paper tackles the challenge of recovering rewards that are robust to these changing market regimes.
Methods
The authors employ a two-stage pipeline to make the problem tractable:
-
World Models in Latent Space: Raw Limit Order Book (LOB) data is high-dimensional and noisy. The authors use a CNN-based autoencoder to compress this data into a 12-dimensional latent vector ($z$). They then use a Recurrent Neural Network (RNN) combined with a Mixture Density Network (MDN) to predict future states. This "World Model" acts as a simulator where agents can be trained safely.
Figure 2: Model for prediction of latent space transitions. Our RL processes are run in the latent state space. -
Adversarial Learning: They compare two adversarial approaches:
- GAIL (Generative Adversarial Imitation Learning): Directly learns a policy to mimic the expert.
- AIRL (Adversarial Inverse Reinforcement Learning): A more advanced approach that attempts to "disentangle" the agent's reward from the environment's dynamics.
By training AIRL within the latent space of the World Model, the authors aim to recover a reward function that represents the expert's intent, making it transferable to new market conditions.
Datasets
The study uses real-world tick data from Tencent Holdings Ltd. (HK-700) traded on the Hong Kong stock exchange.
- Training Period: 60 trading days (January to March 2018).
- Testing Period: 20 trading days (April 2018).
- State Representation: Each state consists of 10 consecutive LOB data ticks across 3 levels of the book, capturing features like bid/ask prices and volumes.
Key Insights and Ideas
- The Power of Compression: Training RL agents on raw LOB data is computationally expensive. By moving to a 12D latent space, the authors created a "distilled" version of the market that retains essential transition dynamics while being much easier for algorithms to digest.
- Disentanglement Matters: The "aha moment" is that AIRL outperforms GAIL specifically because it recovers a reward function, not just a policy. In finance, the "rules" of the market (dynamics) are non-stationary. AIRL’s ability to separate the agent's goals from the environment's current state allows it to adapt better when volatility or price trends shift.
- Latent Space Transferability: The results confirm that policies and rewards learned in a latent-space simulation can be successfully transferred back to "real" data regimes (the April test set) and remain profitable.
Key Results
The experiments used three types of experts: Advantage Actor-Critic (A2C), Double DQN, and Policy Gradient (PG). In every single case, AIRL outperformed GAIL in terms of cumulative reward recovered.
| Expert Type | Expert Avg. Return | AIRL Avg. Return | GAIL Avg. Return |
|---|---|---|---|
| A2C | 5047.3 | 2433.0 | 1563.1 |
| DQN | 4177.4 | 2151.1 | 1574.5 |
| PG | 1483.3 | 1267.7 | 1029.1 |
As shown in the training curves, AIRL's performance continues to climb as iterations increase, while GAIL often plateaus earlier, struggling to capture the nuances of the expert's strategy in a changing environment.
Limitations and Future Work
- Performance Gap: While AIRL beats GAIL, it still only recovers about 50% of the total cumulative reward of the best experts (like A2C). There is still a significant "information loss" during the recovery process.
- Expert Optimality: The model assumes the observed trajectories come from an optimal expert. In real-world applications, "expert" data might be noisy or sub-optimal, which could confuse the adversarial discriminator.
- Future Directions: The authors suggest exploring more complex LOB features and testing the robustness of these recovered rewards across even longer time horizons and more diverse market regimes.
Summary updated