TL;DR
This paper introduces TRIALS, a unified framework for pair trading that replaces the traditional two-step "select then trade" pipeline with a Feudal Hierarchical Reinforcement Learning (HRL) approach. By jointly optimizing pair selection and trading execution, the model overcomes information bottlenecks and achieves significantly higher Sharpe ratios and annualized returns on both U.S. and Chinese stock markets.
What Problem Does This Paper Address?
Pair trading is a classic statistical arbitrage strategy: you find two correlated assets (like Google and Facebook), wait for their price spread to diverge from the historical norm, and then "bet" on them returning to that norm by longing the undervalued asset and shorting the overvalued one.
Traditionally, this is done in two isolated stages:
- Pair Selection: Using statistical tests (like cointegration) to find assets that move together.
- Trading: Applying fixed rules or a separate agent to execute trades.
The authors argue this decoupling is flawed. Selection metrics are often "target irrelevant"—a pair might have a perfect historical correlation but offer zero trading opportunities in the future. Conversely, trading agents trained on a single pre-selected pair often overfit and fail to generalize to broader market dynamics.
Methods
The authors propose a "feudal" hierarchical structure to bridge this gap, where a high-level "Manager" and a low-level "Worker" operate in tandem.
- The Manager (High-Level Policy): Acts as a contextual bandit. It observes the historical price features of all available assets during a "formation period" and selects the optimal pair to trade. Its reward is the actual profit generated by the worker during the subsequent "trading period."
- The Worker (Low-Level Policy): Operates as a Partially Observable Markov Decision Process (POMDP). Once the Manager picks a pair, the Worker executes a series of actions (Long, Short, or Clear). It is trained using an intrinsic reward based on performance during the formation period, which forces it to learn general profitable patterns rather than just memorizing one pair.
- Architecture: Both levels utilize Bi-directional GRUs (Bi-GRU) to capture temporal dependencies in price data, coupled with a temporal attention mechanism to focus on the most salient historical price movements.
Datasets
The model was evaluated on two major real-world datasets spanning 21 years (2000–2020):
- S&P 500 (U.S.): 150 stocks with 5,284 trading days.
- CSI 300 (China): 300 stocks with 5,088 trading days.
The authors randomly split these stocks into five non-overlapping sub-datasets to test the model's generality across different market sectors.
Key Insights and Ideas
- Trading-Aware Selection: By making the Manager's reward dependent on the Worker's trading profit, the system learns to select pairs that aren't just "correlated" in a vacuum but are actually profitable to trade.
- Generalization through Hierarchy: Because the Worker is exposed to various pairs selected by the Manager during training, it learns "common profitable patterns" of mean reversion rather than overfitting to the quirks of a single asset pair.
- Temporal Salience: The attention mechanism reveals that the model places higher weight on recent price features (e.g., for AAPL), confirming that recent market state is more critical for immediate arbitrage opportunities.
Key Results
TRIALS consistently outperformed traditional statistical methods (GGR, Cointegration, Correlation) and standalone RL trading agents (Wang et al.).
- Sharpe Ratio (SR): On the S&P 500, TRIALS achieved an average SR of 1.84, compared to 1.18 for the best RL baseline and negative ratios for traditional statistical selection methods.
- Annualized Return (AR): TRIALS delivered a 0.50 AR on the S&P 500, more than double the best baseline (0.21).
- Risk Management: Despite the higher returns, TRIALS maintained a low Maximum Drawdown (MDD) of -0.09, suggesting the strategy effectively hedges market risk as intended for a market-neutral strategy.
Limitations and Future Work
While TRIALS shows strong performance using fundamental price data (open, close, volume), the authors note that financial markets are influenced by much more. Future work could involve:
- Multi-modal Data: Integrating natural language processing (NLP) to digest news sentiment and social media.
- Macro Factors: Incorporating macroeconomic variables (interest rates, inflation) into the state representation.
- Advanced Representations: Exploring more sophisticated asset embedding techniques to better capture "multi-hop" relationships between stocks (e.g., industry spillover effects).
Summary updated