An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization

Siyu Lin, Peter A. BelingIJCAI 2020 · DOI

TL;DR

This paper introduces an end-to-end framework for optimal trade execution using Proximal Policy Optimization (PPO). It leverages raw level-2 limit order book data directly, eliminating the need for manual feature engineering, and proposes a novel sparse reward function that simplifies training while achieving superior performance compared to existing methods.

What Problem Does This Paper Address?

In today's financial markets, electronic trading dominates, and optimizing trade execution – minimizing costs while trading a specific volume within a set timeframe – is crucial for brokerage firms and investors. Traditional methods like TWAP (Time-Weighted Average Price) and VWAP (Volume-Weighted Average Price) are simple but lack adaptability and learning capabilities. More advanced approaches, such as those based on dynamic programming or reinforcement learning (RL), often rely on manually engineered features, which are time-consuming to develop and tune, and can be prone to overfitting. This paper aims to bridge this gap by creating an adaptive, end-to-end RL framework that learns optimal trading strategies directly from raw market data without manual feature engineering, using a simplified reward mechanism.

Methods

The core of the proposed framework is the Proximal Policy Optimization (PPO) algorithm, a popular policy gradient method known for its stability and sample efficiency. The authors adapt PPO for optimal trade execution by:

  • End-to-End Learning: Instead of relying on handcrafted features, the framework directly uses raw level-2 limit order book (LOB) data (top 5 bid/ask prices and volumes) along with private state information (remaining inventory and elapsed time).
  • Neural Network Architectures: Two primary architectures are explored:
  • FCN + LSTM: A fully-connected network (FCN) processes the current state, and its output is fed into a Long Short-Term Memory (LSTM) network to capture temporal dependencies in the market data.
  • FCN + Stacked Inputs: An FCN processes stacked LOB information from recent time steps, allowing it to implicitly learn temporal patterns.
  • Sparse Reward Function: A key innovation is the use of a sparse reward signal. Instead of immediate rewards based on metrics like Implementation Shortfall (IS), the agent receives a reward only at the end of each trading episode. This reward reflects the agent's performance relative to a TWAP baseline, simplifying the reward structure and reducing the risk of overfitting compared to shaped rewards.
  • Zero Ending Inventory Constraint: The framework incorporates a mechanism to ensure all shares are liquidated by the end of the trading period, a critical real-world constraint.

Datasets

The experiments were conducted using one year of millisecond Trade and Quote (TAQ) data from the NYSE, specifically from January 1st, 2018, to December 31st, 2018, downloaded from WRDS. This data was used to reconstruct the LOB, aggregating information at 5-second intervals. The data was split into training (January-September) and testing (October-December) sets. The authors also mention using Facebook data for hyperparameter tuning.

Key Insights and Ideas

  • Eliminating Feature Engineering: The most significant insight is that an end-to-end RL system can learn effective trade execution strategies directly from raw LOB data, bypassing the laborious and error-prone process of manual feature engineering. This makes the system more adaptable and easier to deploy.
  • Simplicity of Sparse Rewards: The paper demonstrates that a simple, sparse reward signal (end-of-episode relative performance) can be as effective, if not more so, than complex shaped rewards or immediate IS rewards. This simplifies the RL problem and potentially reduces overfitting.
  • LSTM for Temporal Patterns: The LSTM architecture proved effective in capturing the temporal dynamics of the market, leading to better performance in identifying optimal trading sequences.
  • PPO's Robustness: PPO's inherent stability and sample efficiency make it a suitable algorithm for this complex financial task, outperforming other DRL methods in terms of both execution cost and trading speed.

Key Results

The proposed PPO-based framework, particularly the PPO-LSTM and PPO-Stack variants, significantly outperformed traditional benchmarks (TWAP, VWAP, AC) and several other Deep Reinforcement Learning (DRL) models (DDQN, DQN, DQN Sparse) across 14 US equities.

  • Performance: The PPO models achieved better execution quality, indicated by lower Implementation Shortfall (IS) values (reported as mean AIS, where lower is better).
  • Speed: The PPO models also tended to complete trades faster, reducing timing risk.
  • Stability: While DDQN showed good performance on some stocks, it exhibited significant oscillations, which can be detrimental in real-world trading. The PPO models demonstrated more stable learning curves.
  • Specific Numbers: Table 2 shows that PPO-LSTM and PPO-Stack generally achieved the lowest mean AIS values across most stocks, with PPO-LSTM often showing the best performance. For example, on GOOG, PPO-LSTM had a mean AIS of -1.58, significantly better than TWAP (0.00) or DDQN (16.65).

Limitations and Future Work

  • Assumptions: The study relies on several assumptions:
  • Temporary Market Impact: Actions have only a temporary impact, and the market quickly returns to equilibrium. This is a common assumption in RL for trading but may not hold in all scenarios.
  • No Commissions/Fees: Transaction costs like commissions and exchange fees were ignored, which is a simplification for institutional investors.
  • Market Resilience: The assumption of market resilience is crucial for training on historical data without accounting for permanent market impact.
  • Fast Access: Assumes direct and fast exchange access with no order arrival delays.
  • Future Work: The authors plan to relax the assumption of independent agent actions. Future research will focus on modeling the interactions of multiple DRL agents and their collective market decisions, moving towards a more realistic multi-agent trading environment.

Summary updated

BibTeX
@inproceedings{lin2020end,
  author = {Siyu Lin and Peter A. Beling},
  title = {{An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/627},
}
Figure