TL;DR
This paper adapts adversarial inverse reinforcement learning (AIRL) to a simulated financial market environment derived from real limit order book data. The goal is to recover an agent's reward function from its trading behavior, even when market dynamics are volatile and non-stationary, demonstrating AIRL's robustness over standard imitation learning methods.
What Problem Does This Paper Address?
Reinforcement learning (RL) agents typically learn by trial-and-error, which is often infeasible in high-stakes, real-time environments like financial markets due to the risk of costly exploration. Inverse Reinforcement Learning (IRL) offers a solution by inferring an agent's reward function from expert demonstrations. However, traditional IRL struggles with non-stationary environments and the inherent ambiguity of inferring rewards. This paper tackles the challenge of applying advanced IRL techniques, specifically Adversarial Inverse Reinforcement Learning (AIRL), to a realistic financial market setting. The key problem is to learn reward functions that are robust to changing market conditions and can be learned efficiently within a simulated latent space of the market, avoiding direct interaction with the risky real market.
Methods
The core of the paper's methodology involves adapting Adversarial Inverse Reinforcement Learning (AIRL) [8] to a simulated financial market. This approach uses a generator-discriminator framework, similar to Generative Adversarial Networks (GANs) [10].
- World Models for Financial Markets: The authors leverage a "World Model" approach [13] to create a simulated environment. This involves:
- An auto-encoder to compress high-dimensional Limit Order Book (LOB) data into a lower-dimensional latent space representation.
- A Recurrent Neural Network (RNN) combined with a Mixture Density Network (MDN) to learn the transition dynamics within this latent space. This allows the model to predict future states based on current states and actions.
- Expert Agent Training: Three different RL algorithms (Advantage Actor-Critic (A2C) [14], Double DQN [2], and Policy Gradient [15]) are trained within this latent space world model to act as "expert traders." Their state-action trajectories are collected.
- Adversarial Inverse Reinforcement Learning (AIRL):
- The generator in AIRL learns to produce state-action pairs that mimic the expert demonstrations.
- The discriminator tries to distinguish between real expert trajectories and those generated by the generator.
- Crucially, AIRL aims to recover a reward function that is robust to variations in the environment's dynamics.
- Benchmark: Generative Adversarial Imitation Learning (GAIL) [11] is used as a benchmark to compare performance against AIRL.
Datasets
- Training Data: Real Limit Order Book (LOB) time series data for Tencent Holdings Ltd. (HK-700) from sixty trading days between January and March 2018.
- Testing Data: LOB data from the subsequent twenty trading days in April 2018.
- Benchmark: The paper uses an OpenAI Gym structure [23] adapted for financial environments, building upon existing work [12] that uses LOB data.
Key Insights and Ideas
- Latent Space Simulation is Viable for Financial IRL: The paper demonstrates that complex financial market dynamics can be effectively simulated in a latent space, allowing advanced RL and IRL algorithms to be trained without direct, risky interaction with real market data.
- AIRL's Robustness Advantage: AIRL significantly outperforms GAIL in recovering cumulative rewards. This is attributed to AIRL's ability to learn reward functions that are robust to the non-stationary and volatile nature of financial markets, a key challenge that GAIL struggles with.
- Expert Performance Matters: The performance of the recovered rewards is influenced by the underlying expert agent. The A2C agent, which showed the best performance among the trained experts, also led to better results when its demonstrations were used for AIRL and GAIL.
- Disentangled Rewards for Volatility: The core idea of AIRL—learning disentangled rewards that are robust to environmental dynamics—is particularly well-suited for financial markets, which are inherently volatile and non-stationary.
Key Results
- AIRL Outperforms GAIL: In all three experimental setups (training with A2C, DQN, and PG experts), AIRL consistently achieved higher cumulative rewards than GAIL when evaluated on the testing data.
- For A2C experts, AIRL recovered 2433.0 reward (vs. GAIL's 1563.1).
- For DQN experts, AIRL recovered 2151.1 reward (vs. GAIL's 1574.5).
- For PG experts, AIRL recovered 1267.7 reward (vs. GAIL's 1029.1).
- AIRL Rewards are Robust: The paper highlights that the rewards learned by AIRL from the training model are robust to the variable dynamics observed between the training and testing periods, leading to its superior performance on the test data.
Limitations and Future Work
- Limitations:
- The study focuses on a specific financial instrument (Tencent Holdings Ltd. stock) and a limited time frame.
- While the latent space model captures dynamics, it's still a simplification of the full, complex real-world market.
- The paper acknowledges that the "opinions expressed are those of the authors, and do not necessarily reflect the view of JP Morgan."
- Future Work:
- The authors suggest that the demonstrated robustness of AIRL makes it an attractive method for further study in financial markets, particularly for learning from experts in new market regimes.
- Further exploration of adapting AIRL to more complex financial instruments and market structures is implied.
Summary updated