Abstract
Deep Reinforcement Learning (DRL) in financial markets faces significant challenges due to non-stationary data and the inherent bias-variance trade-off in off-policy learning. We present Safe-FinRL, a framework designed to stabilize high-frequency stock trading. By partitioning volatile market data into "near-stationary" segments and introducing Trace-SAC—an algorithm combining Soft Actor-Critic with multi-step "retrace" operators—the framework significantly reduces bias and variance. Evaluation on high-frequency cryptocurrency data demonstrates that Safe-FinRL achieves more stable policy improvement and superior profitability compared to standard single-step baselines.
Introduction
Applying DRL to financial markets is notoriously difficult due to two primary hurdles. First, non-stationarity means that statistical properties of price movements shift constantly, making strategies learned on past data quickly irrelevant. Second, the bias-variance trade-off in off-policy learning creates a tension between low-variance but high-bias single-step updates and low-bias but high-variance multi-step updates. Most existing DRL trading models ignore these stability issues, leading to poor convergence in real-world, high-frequency scenarios.
Methodology
Environment Separation
To combat non-stationarity, Safe-FinRL avoids training a single model on a long, chaotic time series. Instead, the data is divided into independent, shorter "near-stationary" environments. This approach assumes that over a short window (e.g., 5 days), market behavior is stable enough for an agent to learn meaningful patterns. Each environment is further split into training and validation/testing phases to ensure robust policy evaluation.
Trace-SAC Algorithm
The core algorithmic contribution is Trace-SAC, which integrates the General Retrace operator into the Soft Actor-Critic (SAC) framework. SAC provides a maximum-entropy objective that encourages exploration, while the Retrace operator allows for multi-step return estimation. By using importance sampling "traces" to correct for distribution shifts between the current policy and the replay buffer, Trace-SAC enables multi-step learning (reducing bias) without the explosive variance typically associated with long-horizon lookaheads.
Experimental Setup
The study utilizes high-frequency cryptocurrency data from Binance (BTC-USDT swap) over a 20-day period in March 2022. The data is processed at minute-by-minute granularity, yielding approximately 28,800 samples. The model architecture employs a Long Short-Term Memory (LSTM) network to encode 118 technical features, followed by fully connected layers to output Q-values and a Gaussian policy for trading actions.
Results and Discussion
Experimental results across four distinct environments demonstrate the stability and efficacy of the Trace-SAC approach. As shown in the reward plots below, trace-based methods (such as Retrace and Q-lambda) generally achieve higher average rewards and faster convergence than the uncorrected or standard single-step baselines.
Key findings include:
- Stability: Trace-SAC demonstrated significantly lower variance in value estimation compared to vanilla SAC.
- Profitability: Safe-FinRL achieved positive cumulative log returns across all test environments, outperforming market benchmarks in Env0 and Env2.
- Trace Comparison: "Conservative" trace methods, which damp the influence of old data when it deviates significantly from the current policy, proved essential for safe value estimation in noisy high-frequency markets.
Conclusion
Safe-FinRL addresses the fundamental instabilities of applying DRL to high-frequency trading. By combining local stationarity assumptions with multi-step trace operators, the framework provides a more reliable path to policy convergence. Future work will focus on automating the detection of market regime shifts and improving the interpretability of the agent's trading decisions.
Summary updated