Introduction
In algorithmic trading, a common paradigm involves a two-stage process: predicting future price movements via supervised learning and executing trades based on those predictions. However, this approach often struggles in "thin" or illiquid markets characterized by high transaction costs and limited data. A primary obstacle is the lack of generalization; deep learning models often overfit on sparse financial datasets, leading Reinforcement Learning (RL) agents to rely on "perfect" but unrealistic signals. This paper introduces a "data cross-segmentation" technique designed to expose the RL agent to the inherent noise of real-world predictions during training, thereby enhancing its robustness.
Methodology: A Two-Stage Pipeline
The proposed framework decouples price prediction from strategy execution to manage complexity and improve stability.
Stage 1: Price Prediction
The first stage utilizes a Gated Recurrent Unit (GRU) neural network to process 30-day windows of historical features, including RSI, volatility, and moving averages. Rather than predicting exact prices, the model outputs the probability of upward or downward movement. To mitigate the high variance typical of financial time-series, an ensemble of GRU models is employed, acting as a low-pass filter for the subsequent stage.
Stage 2: Trading Strategy
The second stage features an RL agent using the Proximal Policy Optimization (PPO) algorithm. This agent receives the signals from Stage 1 and determines wealth allocation across the portfolio. The reward function is specifically tuned to maximize returns while penalizing negative returns more heavily to maintain a favorable Sharpe ratio.
Data Cross-Segmentation
The core innovation of this work is the data cross-segmentation strategy. Standard training often uses the same data for both the predictor and the RL agent, which allows the agent to exploit the predictor's overfitting. To prevent this, the authors split the training data into disjoint sets, train_1 and train_2.
The Stage 1 predictor is trained on train_1, and its predictions for train_2 are then used to train the Stage 2 RL agent. Since the predictor has not seen train_2 during its optimization, the signals it generates for this set contain realistic errors and noise. This forces the RL agent to learn a strategy that generalizes to imperfect real-world data.
Experimental Setup and Datasets
The study leverages over 20 years of equity data from Bursa Malaysia (2000–2021). Features include OHLC prices, trading volume, and engineered indicators like moving averages (5 to 200 days) and the Relative Strength Index (RSI). The data is partitioned into training (2000–2018), validation (2018–2019), and testing (2020–2021) segments to evaluate performance under recent market conditions.
Key Results and Insights
The experiments yielded several critical findings:
- Improved Generalization: The "Quarterly" cross-segmentation method, which rotates training sets periodically, consistently outperformed models trained on the full dataset. While the "Full" model's performance typically degraded over time due to overfitting, cross-segmented models showed sustained improvement.
- Performance Benchmarks: The RL-based strategy achieved a Sharpe ratio of approximately 1.0 in zero-cost environments. Crucially, it significantly outperformed a standard "buy and hold" strategy, which yielded a negative Sharpe ratio (-0.05) during the test period.
- Risk-Averse Learning: By weighting negative returns more heavily in the reward function, the agent learned to prioritize stability, a necessity in high-friction markets.
Conclusion
By introducing data cross-segmentation, this research addresses the fundamental "fragile agent" trap in RL-based trading. Exposing the strategy agent to noisy, out-of-sample predictions during training ensures that it develops robust execution policies. Future work will explore longer prediction horizons and more advanced architectures, such as Transformers, to further capture complex market dependencies.
Summary updated