Abstract
Financial trading aims to build profitable strategies to make wise investment decisions in the financial market. It has attracted interests in the machine learning community for a long time. This paper proposes to trade financial assets automatically using feature preprocessing skills and Recurrent Reinforcement Learning (RRL) algorithm. The strategy starts from technical indicators extracted from assets' market information. Then these technical indicators are preprocessed by Principal Component Analysis (PCA) and Discrete Wavelet Transform (DWT) and eventually inputted to the RRL algorithm to do the trading. The extensive empirical evidence shows that the proposed strategy is not only effective and robust in its performance, but also can mitigate the drawbacks underlying the initial trading using RRL.
Introduction
Recurrent Reinforcement Learning (RRL) is a prominent approach in algorithmic trading due to its ability to optimize risk-adjusted returns, such as the Sharpe Ratio, directly. However, standard RRL implementations often suffer from high sensitivity to hyperparameters, vulnerability to market noise, and an inability to adapt to structural market shifts, leading to significant "precipitous drops" in profit. This research addresses these issues by focusing on the quality of input data. Rather than increasing model complexity, the authors propose a robust preprocessing pipeline to "clean the signal" before it reaches the reinforcement learning agent.
Methodology
The proposed system, PCA&DWT RRL, utilizes a multi-stage pipeline to transform raw market data into actionable trading signals:
- Feature Extraction: Eleven technical indicators (e.g., MACD, RSI) are extracted and Z-score normalized.
- Dimensionality Reduction (PCA): Principal Component Analysis is applied to retain components explaining 95% of the variance, reducing redundancy and focusing on significant signals.
- Denoising (DWT): Discrete Wavelet Transform (Haar wavelets) with soft thresholding is used to strip away high-frequency noise while preserving underlying price trends.
- Recurrent Reinforcement Learning: The cleaned features are fed into an RRL model. Unlike traditional predictive models, the RRL outputs a trading position (Long or Short), accounting for previous positions to incorporate transaction costs. The model is trained via batch gradient ascent to maximize the Sharpe Ratio.
Experimental Results
The strategy was evaluated on the NYSE Composite Index, Exxon Mobil (XOM), and Corn Futures (ZCF) from 2001 to 2021. This period includes major market disruptions like the 2008 financial crisis and the 2020 pandemic.
As shown in the performance comparison above, the PCA&DWT RRL method significantly outperforms benchmarks. In the NYSE dataset, it achieved a net profit of $32,286.96, compared to $10,504.62 for a Buy & Hold strategy. Notably, the vanilla RRL approach (TI RRL) resulted in a net loss.
The proposed method maintained an Annualized Sharpe Ratio (ASR) above 1.0 across all datasets, demonstrating superior risk-adjusted returns. Furthermore, the Maximum Drawdown (MDD) was drastically reduced; for instance, the NYSE MDD was only 0.12 for the proposed method versus 10.35 for vanilla RRL.
Discussion and Conclusion
A key insight from this study is that denoising via DWT is more impactful for performance than dimensionality reduction via PCA. By removing "static" from financial time series, the RRL model becomes more stable and less prone to being "faked out" by noisy price spikes. This preprocessing approach effectively mitigates the "precipitous drop" phenomenon common in standard RRL.
Future work may explore adding a "Neutral" position for better risk management during extreme volatility and further automating hyperparameter selection. Overall, the integration of PCA and DWT provides a powerful framework for enhancing reinforcement learning in complex financial environments.
Summary updated