LSTM-DDPG for Trading with Variable Positions

Zhichao Jia, Qiang Gao, Xiaohong PengSensors 2021 · DOI

TL;DR

This paper introduces LSTM-DDPG, a novel deep reinforcement learning approach for financial trading that uniquely allows for variable position sizing. By combining LSTM for market state feature extraction and DDPG for decision-making, the model demonstrates superior performance in balancing profit and risk compared to fixed or discrete position sizing strategies.

What Problem Does This Paper Address?

Traditional machine learning approaches to trading often struggle with the inherent non-stationarity of financial markets. While deep reinforcement learning (DRL) has shown promise, existing methods typically focus on determining the direction of a trade (long or short) but often assume a fixed or limited set of position sizes. This simplification overlooks a crucial aspect of trading: dynamically adjusting the amount of capital invested based on market conditions to maximize profit while mitigating risk. The paper addresses this gap by proposing a DRL framework that can learn to control both the direction and the precise, variable size of a trading position.

Methods

The core of the proposed solution is the LSTM-DDPG model. This framework integrates two key components:

  • Long Short-Term Memory (LSTM) Network: Used to process historical market data (specifically, the closing prices of the past 20 trading days) to extract relevant market state features. LSTMs are well-suited for capturing long-term dependencies in sequential data, which is crucial for understanding market trends.
  • Deep Deterministic Policy Gradient (DDPG) Framework: This DRL algorithm is responsible for making trading decisions. DDPG is an actor-critic method that learns a deterministic policy to output continuous actions. In this context, the action is a single real number between -1 and +1. The sign of the action determines the position direction (positive for long, negative for short), and its absolute value represents the variable position size as a percentage of total capital.

The trading process is modeled as a Partially Observable Markov Decision Process (POMDP), acknowledging that the true market state is not fully known to the trading agent. The paper also explores two reward functions: a simple profit reward and a more sophisticated Differential Sharpe Ratio (DSR) reward, which explicitly considers both profit and risk (volatility).

Datasets

The model was trained and evaluated on the China IF300 index futures data. This dataset comprises daily closing prices of the top 300 stocks from the Shanghai and Shenzhen exchange centers over an 18-year period (July 2002 to June 2020). The data was split into training (July 2002 - June 2014), validation (July 2014 - June 2017), and testing (July 2017 - June 2020) sets.

Key Insights and Ideas

  • Variable Position Sizing is Crucial: The research strongly suggests that dynamically adjusting position sizes based on market conditions is key to improving trading performance. This goes beyond simply deciding whether to go long or short.
  • DSR Rewards Lead to Smoother, More Robust Performance: While a pure profit reward might yield higher raw profits, the DSR reward function results in a smoother profit curve and better overall risk-adjusted performance, indicating a more stable and sustainable trading strategy.
  • LSTM-DDPG Adapts to Market Volatility: The model's ability to adjust position size allows it to capitalize on strong trends (larger positions) and reduce exposure during volatile or uncertain periods (smaller or zero positions), effectively managing risk.

Key Results

  • Superior Performance with Variable Positions: The LSTM-DDPG model with variable position sizing achieved a higher Sharpe ratio (0.328) compared to fixed (0.215) and few-level (0.265) position strategies when using the DSR reward function.
  • Profit vs. Risk Trade-off: For the profit reward function, the model achieved a total return rate of 42.5% with a maximum drawdown of 15.3%. With the DSR reward function (without leverage), it achieved a total return rate of 29.8% and a significantly lower maximum drawdown of 9.5%.
  • Leverage Enhances DSR Performance: When leverage was applied, the DSR reward function yielded a total return rate of 59.8% with a maximum drawdown of 18.1%, demonstrating its potential to amplify gains while managing risk.
  • DSR Outperforms Profit for Risk-Adjusted Returns: Quantitatively, the DSR reward function consistently provided a better Sharpe ratio, indicating a more favorable balance between profit and risk.

Limitations and Future Work

  • Limited Input Features: The current model primarily uses closing prices. The authors suggest that incorporating other features like high price, low price, and trading volume could further enhance performance.
  • Ignoring Previous Actions: The paper notes that transaction fees and slippage are affected by previous actions, which are not explicitly considered in the current model's decision-making process. Future work could incorporate the previous trading action into the actor network's input.
  • Market Expansion: The experiments were conducted on Chinese stock index futures. The authors propose extending the model's evaluation to other markets such as stocks, commodity futures, and foreign exchange futures to demonstrate broader applicability.

Summary updated

Abstract

In recent years, machine learning for trading has been widely studied. The direction and size of position should be determined in trading decisions based on market conditions. However, there is no research so far that considers variable position sizes in models developed for trading purposes. In this paper, we propose a deep reinforcement learning model named LSTM-DDPG to make trading decisions with variable positions. Specifically, we consider the trading process as a Partially Observable Markov Decision Process, in which the long short-term memory (LSTM) network is used to extract market state features and the deep deterministic policy gradient (DDPG) framework is used to make trading decisions concerning the direction and variable size of position. We test the LSTM-DDPG model on IF300 (index futures of China stock market) data and the results show that LSTM-DDPG with variable positions performs better in terms of return and risk than models with fixed or few-level positions. In addition, the investment potential of the model can be better tapped by the reward function of the differential Sharpe ratio than that of profit reward function.
BibTeX
@inproceedings{jia2021lstm,
  author = {Zhichao Jia and Qiang Gao and Xiaohong Peng},
  title = {{LSTM-DDPG for Trading with Variable Positions}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21196571},
}
Figure