Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution

Feiyang Pan, Tongzhe Zhang, Ling Luo, Jia He, Shuoling LiuIJCAI 2022 · DOI

TL;DR

This paper introduces HALOP, a novel two-stage reinforcement learning approach for optimal algorithmic trading execution. HALOP addresses the "continuous-discrete duality" of setting limit prices by first using a generalized continuous policy to narrow down options and then a specialized discrete policy to pick the exact price, leading to improved cost savings and stability.

What Problem Does This Paper Address?

Executing large orders in financial markets efficiently is a critical challenge for traders. The goal is to minimize costs and market impact by breaking down a large order into smaller "child" orders. While market orders execute immediately but at potentially unfavorable prices, limit orders offer price control but may not be filled. A key difficulty in using reinforcement learning (RL) for limit order execution is deciding both the price and volume of these child orders. Specifically, setting limit prices presents a "continuous-discrete duality": traders think about price movements in percentages (continuous, good for generalization across different stocks), but ultimately must choose prices that align with the market's tick size (discrete, good for specialization to a specific stock's characteristics). Existing RL methods struggle to balance these two aspects effectively, either sacrificing generalization or precise control.

Methods

The paper proposes HALOP (Hybrid Action-space Limit Order Placement), a two-stage RL framework built upon the Proximal Policy Optimization (PPO) algorithm.

  • Stage 1: Generalization (Discretized Gaussian Policy): This stage uses a generalized policy that takes standardized public market data as input. It employs a discretized Gaussian policy to output a discrete percentage change in price. This allows the agent to learn general patterns across various stocks without being overly sensitive to specific price levels or liquidity. The output of this stage scopes a subset of potential discrete tick-based actions for the next stage.
  • Stage 2: Specialization (Gaussian-and-Softmax Policy): This stage takes the scoped action subset from Stage 1 and uses a fine-grained discrete policy (specifically, a Gaussian-and-Softmax policy) to select the exact limit price in ticks. This stage incorporates private state information (like remaining inventory and time) and raw, unstandardized market data, enabling it to specialize for the unique characteristics of each stock.

The core novelty lies in this hierarchical approach, leveraging a discretized Gaussian policy for broad generalization and a Gaussian-and-Softmax policy for precise, specialized action selection.

Datasets

The experiments were conducted using historical high-frequency transaction data for stocks in the CSI 300 index in the China A-shares market. The data from January 2010 to December 2019 was used for training, and data from January 2020 to June 2020 was used for testing.

Key Insights and Ideas

  • The "Continuous-Discrete Duality" is Key: The paper effectively frames the limit price setting problem as a trade-off between continuous percentage-based price movements (for generalization) and discrete tick-based prices (for specialization).
  • Two-Stage Approach Solves the Duality: HALOP's hierarchical structure is designed to tackle this duality directly. Stage 1 provides broad generalization, while Stage 2 offers fine-grained specialization.
  • Discretized Gaussian Policy for Generalization: Using a discretized Gaussian policy in Stage 1 allows for learning robust price-setting strategies that are less sensitive to specific stock characteristics, improving generalization.
  • Specialization Improves Performance: The experiments show that Stage 2's specialization significantly boosts both excess return and stability, particularly for "penny stocks" (low-priced, less liquid stocks).
  • HALOP Outperforms Market Orders and Vanilla RL: The proposed method consistently beats market orders and shows substantial improvements over standard PPO with Gaussian or Softmax policies.

Key Results

  • HALOP achieved an average excess return of 4.41 bps over the TWAP-with-market-order benchmark, with a PnL of 2.98 and a t-value of 123.2, indicating statistically significant improvement.
  • Compared to PPO-Gaussian and PPO-Softmax, HALOP demonstrated superior performance in both excess return and PnL, while also exhibiting much better stability (lower Std).
  • HALOP Stage-1 alone significantly improved upon vanilla PPO methods, highlighting the benefit of its discretized Gaussian policy for generalization.
  • The grouping study showed that PPO-Gaussian favored high-priced stocks, while HALOP Stage-1 generalized well across all price groups. HALOP (two-stage) further improved results for low-priced stocks, demonstrating effective specialization.

Limitations and Future Work

  • Simulation Environment: Experiments were conducted in a simulated environment. While efforts were made to make it realistic (e.g., communication delay, order size limits), real-world market dynamics can be more complex.
  • Focus on Price: The paper primarily focuses on setting limit prices, assuming a predefined volume schedule. While it mentions the method can work with volume-oriented methods, integrating end-to-end price and volume optimization could be a future direction.
  • Future Work: The authors suggest that HALOP offers a reference for applying RL in real-world scenarios where generalization and specialization trade-offs are important. Further exploration of integrating volume optimization and testing in live trading environments would be valuable.

Summary updated

BibTeX
@inproceedings{pan2022learn,
  author = {Feiyang Pan and Tongzhe Zhang and Ling Luo and Jia He and Shuoling Liu},
  title = {{Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/543},
}
Figure