Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning

Weiguang Han, Boyi Zhang, Qianqian Xie, Min Peng, Yanzhao Lai, Jimin Huang

TL;DR

This paper introduces TRIALS, a unified framework for pair trading that replaces the traditional two-step "select then trade" pipeline with a Feudal Hierarchical Reinforcement Learning (HRL) approach. By jointly optimizing pair selection and trading execution, the model overcomes information bottlenecks and achieves significantly higher Sharpe ratios and annualized returns on both U.S. and Chinese stock markets.

What Problem Does This Paper Address?

Pair trading is a classic statistical arbitrage strategy: you find two correlated assets (like Google and Facebook), wait for their price spread to diverge from the historical norm, and then "bet" on them returning to that norm by longing the undervalued asset and shorting the overvalued one.

Traditionally, this is done in two isolated stages:

  1. Pair Selection: Using statistical tests (like cointegration) to find assets that move together.
  2. Trading: Applying fixed rules or a separate agent to execute trades.

The authors argue this decoupling is flawed. Selection metrics are often "target irrelevant"—a pair might have a perfect historical correlation but offer zero trading opportunities in the future. Conversely, trading agents trained on a single pre-selected pair often overfit and fail to generalize to broader market dynamics.

Methods

The authors propose a "feudal" hierarchical structure to bridge this gap, where a high-level "Manager" and a low-level "Worker" operate in tandem.

Figure 1: The hierarchical framework for pair trading.
  • The Manager (High-Level Policy): Acts as a contextual bandit. It observes the historical price features of all available assets during a "formation period" and selects the optimal pair to trade. Its reward is the actual profit generated by the worker during the subsequent "trading period."
  • The Worker (Low-Level Policy): Operates as a Partially Observable Markov Decision Process (POMDP). Once the Manager picks a pair, the Worker executes a series of actions (Long, Short, or Clear). It is trained using an intrinsic reward based on performance during the formation period, which forces it to learn general profitable patterns rather than just memorizing one pair.
  • Architecture: Both levels utilize Bi-directional GRUs (Bi-GRU) to capture temporal dependencies in price data, coupled with a temporal attention mechanism to focus on the most salient historical price movements.
Figure 2: The feudal hierarchical reinforcement learning framework.

Datasets

The model was evaluated on two major real-world datasets spanning 21 years (2000–2020):

  • S&P 500 (U.S.): 150 stocks with 5,284 trading days.
  • CSI 300 (China): 300 stocks with 5,088 trading days.

The authors randomly split these stocks into five non-overlapping sub-datasets to test the model's generality across different market sectors.

Key Insights and Ideas

  • Trading-Aware Selection: By making the Manager's reward dependent on the Worker's trading profit, the system learns to select pairs that aren't just "correlated" in a vacuum but are actually profitable to trade.
  • Generalization through Hierarchy: Because the Worker is exposed to various pairs selected by the Manager during training, it learns "common profitable patterns" of mean reversion rather than overfitting to the quirks of a single asset pair.
  • Temporal Salience: The attention mechanism reveals that the model places higher weight on recent price features (e.g., for AAPL), confirming that recent market state is more critical for immediate arbitrage opportunities.
Figure 4: The visualization of temporal attentions

Key Results

TRIALS consistently outperformed traditional statistical methods (GGR, Cointegration, Correlation) and standalone RL trading agents (Wang et al.).

Quantitative results comparing TRIALS against baselines across S&P 500 and CSI 300 datasets.
  • Sharpe Ratio (SR): On the S&P 500, TRIALS achieved an average SR of 1.84, compared to 1.18 for the best RL baseline and negative ratios for traditional statistical selection methods.
  • Annualized Return (AR): TRIALS delivered a 0.50 AR on the S&P 500, more than double the best baseline (0.21).
  • Risk Management: Despite the higher returns, TRIALS maintained a low Maximum Drawdown (MDD) of -0.09, suggesting the strategy effectively hedges market risk as intended for a market-neutral strategy.

Limitations and Future Work

While TRIALS shows strong performance using fundamental price data (open, close, volume), the authors note that financial markets are influenced by much more. Future work could involve:

  • Multi-modal Data: Integrating natural language processing (NLP) to digest news sentiment and social media.
  • Macro Factors: Incorporating macroeconomic variables (interest rates, inflation) into the state representation.
  • Advanced Representations: Exploring more sophisticated asset embedding techniques to better capture "multi-hop" relationships between stocks (e.g., industry spillover effects).

Summary updated

Abstract

Pair trading is one of the most effective statistical arbitrage strategies which seeks a neutral profit by hedging a pair of selected assets. Existing methods generally decompose the task into two separate steps: pair selection and trading. However, the decoupling of two closely related subtasks can block information propagation and lead to limited overall performance. For pair selection, ignoring the trading performance results in the wrong assets being selected with irrelevant price movements, while the agent trained for trading can overfit to the selected assets without any historical information of other assets. To address it, in this paper, we propose a paradigm for automatic pair trading as a unified task rather than a two-step pipeline. We design a hierarchical reinforcement learning framework to jointly learn and optimize two subtasks. A high-level policy would select two assets from all possible combinations and a low-level policy would then perform a series of trading actions. Experimental results on real-world stock data demonstrate the effectiveness of our method on pair trading compared with both existing pair selection and trading methods.
BibTeX
@misc{han2023select,
  author = {Weiguang Han and Boyi Zhang and Qianqian Xie and Min Peng and Yanzhao Lai and Jimin Huang},
  title = {{Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning}},
  year = {2023},
  eprint = {2301.10724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.10724},
}
Figure