Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock Trading

Zitao Song, Xuyang Jin, Chenliang LiarXiv 2022 · DOI · arXiv

Abstract

Deep Reinforcement Learning (DRL) in financial markets faces significant challenges due to non-stationary data and the inherent bias-variance trade-off in off-policy learning. We present Safe-FinRL, a framework designed to stabilize high-frequency stock trading. By partitioning volatile market data into "near-stationary" segments and introducing Trace-SAC—an algorithm combining Soft Actor-Critic with multi-step "retrace" operators—the framework significantly reduces bias and variance. Evaluation on high-frequency cryptocurrency data demonstrates that Safe-FinRL achieves more stable policy improvement and superior profitability compared to standard single-step baselines.

Introduction

Applying DRL to financial markets is notoriously difficult due to two primary hurdles. First, non-stationarity means that statistical properties of price movements shift constantly, making strategies learned on past data quickly irrelevant. Second, the bias-variance trade-off in off-policy learning creates a tension between low-variance but high-bias single-step updates and low-bias but high-variance multi-step updates. Most existing DRL trading models ignore these stability issues, leading to poor convergence in real-world, high-frequency scenarios.

Methodology

Environment Separation

To combat non-stationarity, Safe-FinRL avoids training a single model on a long, chaotic time series. Instead, the data is divided into independent, shorter "near-stationary" environments. This approach assumes that over a short window (e.g., 5 days), market behavior is stable enough for an agent to learn meaningful patterns. Each environment is further split into training and validation/testing phases to ensure robust policy evaluation.

Figure 1: The environment partitioning strategy, dividing the time series into four near-stationary segments (Env0-Env3) with dedicated training and validation windows.

Trace-SAC Algorithm

The core algorithmic contribution is Trace-SAC, which integrates the General Retrace operator into the Soft Actor-Critic (SAC) framework. SAC provides a maximum-entropy objective that encourages exploration, while the Retrace operator allows for multi-step return estimation. By using importance sampling "traces" to correct for distribution shifts between the current policy and the replay buffer, Trace-SAC enables multi-step learning (reducing bias) without the explosive variance typically associated with long-horizon lookaheads.

Experimental Setup

The study utilizes high-frequency cryptocurrency data from Binance (BTC-USDT swap) over a 20-day period in March 2022. The data is processed at minute-by-minute granularity, yielding approximately 28,800 samples. The model architecture employs a Long Short-Term Memory (LSTM) network to encode 118 technical features, followed by fully connected layers to output Q-values and a Gaussian policy for trading actions.

Results and Discussion

Experimental results across four distinct environments demonstrate the stability and efficacy of the Trace-SAC approach. As shown in the reward plots below, trace-based methods (such as Retrace and Q-lambda) generally achieve higher average rewards and faster convergence than the uncorrected or standard single-step baselines.

Figure 3: Reward convergence across four test environments (Env0-Env3) comparing different trace operators. Trace-based methods consistently show improved stability and higher cumulative rewards.

Key findings include:

  • Stability: Trace-SAC demonstrated significantly lower variance in value estimation compared to vanilla SAC.
  • Profitability: Safe-FinRL achieved positive cumulative log returns across all test environments, outperforming market benchmarks in Env0 and Env2.
  • Trace Comparison: "Conservative" trace methods, which damp the influence of old data when it deviates significantly from the current policy, proved essential for safe value estimation in noisy high-frequency markets.

Conclusion

Safe-FinRL addresses the fundamental instabilities of applying DRL to high-frequency trading. By combining local stationarity assumptions with multi-step trace operators, the framework provides a more reliable path to policy convergence. Future work will focus on automating the detection of market regime shifts and improving the interpretability of the agent's trading decisions.

Summary updated

BibTeX
@misc{song2022safe,
  author = {Zitao Song and Xuyang Jin and Chenliang Li},
  title = {{Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock Trading}},
  year = {2022},
  eprint = {2206.05910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.05910},
}
Figure