A General Framework on Enhancing Portfolio Management with Reinforcement Learning

Yinheng Li, Junhao Wang, Yijie CaoIccmb 2024 · DOI

TL;DR

This paper introduces a general reinforcement learning (RL) framework for portfolio management that addresses practical limitations of previous work, such as transaction costs and short selling. By comparing three RL algorithms (PGAC, PPO, ES) on simulated financial data, they demonstrate the potential of RL for enhancing asset allocation strategies.

What Problem Does This Paper Address?

Portfolio management, the art of allocating funds across assets to achieve desired returns while managing risk, is a complex challenge. While Deep Reinforcement Learning (RL) has shown promise in automating this process, existing RL approaches for portfolio management often overlook crucial real-world factors. Specifically, many prior studies fail to account for transaction costs (the fees incurred when buying or selling assets) and restrictions on short selling (selling assets you don't own, a common hedging strategy). These omissions limit the applicability of these RL models to practical financial scenarios. This paper aims to bridge this gap by proposing a more comprehensive RL framework that incorporates these realistic constraints.

Methods

The authors propose a general RL framework for asset management that allows for:

  • Continuous Asset Weights: Agents can allocate any proportion of their portfolio to an asset, rather than being restricted to discrete choices.
  • Short Selling: The framework explicitly supports short selling, providing a more flexible hedging mechanism.
  • Extrinsic Features: The RL agent can utilize a broader set of financial features beyond just asset prices to inform its decisions.

They compare the performance of three distinct RL algorithms within this framework:

  • Policy Gradient Actor-Critic (PGAC): A robust and stable algorithm that uses a separate neural network for value function estimation, helping to avoid local optima.
  • Proximal Policy Optimization (PPO): A popular algorithm known for its state-of-the-art performance, which uses a clipped surrogate objective function to ensure stable policy updates.
  • Evolution Strategy (ES): A gradient-free optimization technique that generates a population of policies and selects the best performers, making it simpler to implement and less reliant on gradient calculations.

The framework also incorporates two types of neural network architectures for the policy network: Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, inspired by previous work.

Datasets

The training data consists of:

  • 4 financial instruments: 3 risky assets (Euro-Bobl Futures (FGBS), Euro-Schatz Futures (FGBM), Euro-Bund Futures (FGBL)) and 1 risk-free asset.
  • 6 financial instruments as features: daily prices of common commodities (gold, copper, crude oil), and daily prices of 5-year and 10-year US treasury bond futures, and the SP500 index.
  • 7300 time steps of daily data from Year 2000 to Year 2020.

The data was selected based on input from domain experts. For evaluation, they randomly sample 10 time spans from the training data, each consisting of 50 days for training and the subsequent 10 days for testing.

Key Insights and Ideas

  • Realism Matters: Incorporating transaction costs and short selling significantly enhances the practical relevance of RL for portfolio management, moving beyond theoretical simulations.
  • Algorithm Diversity: Different RL algorithms (PGAC, PPO, ES) exhibit varying strengths and weaknesses. PPO, particularly with a CNN, shows strong performance, while ES offers a simpler, gradient-free alternative.
  • CNNs Outperform LSTMs (in this context): For the specific EIIE architecture used, CNNs proved more effective than LSTMs in capturing relevant financial signals for portfolio management.
  • Feature Engineering is Key: The framework's ability to incorporate extrinsic features highlights the importance of providing agents with a rich information landscape for better decision-making.
  • ES's Simplicity: Evolution Strategy's advantage lies in its gradient-free nature, simplifying implementation and avoiding the need for explicit value function estimation, which can be challenging in complex financial environments.

Key Results

The paper presents average performance metrics over 10 trials:

  • The CNN-based PPO model achieved the best performance, with the highest average portfolio value (1.044) and the lowest average maximum drawdown (0.30%).
  • On average, the LSTM-based PPO model performed poorly, with a negative Sharpe Ratio (-0.52) and a high maximum drawdown (1.21%).
  • The ES-based model showed promising results, outperforming some RL-based strategies and rule-based baselines with a Sharpe Ratio of 1.08 and a drawdown of 0.43%.
  • All tested RL-based strategies generally outperformed rule-based strategies like "Best Single Asset," "Worst Single Asset," and "Follow the Winner," with the exception of the LSTM-based PPO.
  • The "Fixed Average Weight" strategy also performed reasonably well, highlighting the impact of trading costs.

Limitations and Future Work

  • RL Algorithm Instability: The authors acknowledge that RL algorithms can be unstable across different trials due to high variance in network training, value estimation, and optimization.
  • Comparison Scope: The study did not compare RL with conventional machine learning algorithms, as RL integrates both signal discovery and execution, making direct comparison with signal-generating algorithms unfair.
  • Hyperparameter and Network Exploration: Not all possible network structures, hyperparameter settings, or RL algorithms were explored.
  • Data and Feature Limitations: The experiment was conducted with a specific set of financial data and features, and the findings might not generalize to all markets or time periods.
  • Trading Frequency: The RL agent trades almost every period, which may not be realistic. Future work could explore imposing restrictions on trading frequency and quantity.
  • Generalization: The authors aim to illustrate a generalizable framework rather than identify a state-of-the-art trading algorithm for all scenarios.

Future work could involve exploring more advanced RL algorithms, a wider range of network architectures, and more diverse financial datasets and features. The framework's compatibility with various constraints also opens avenues for more realistic portfolio management simulations.

Summary updated

BibTeX
@inproceedings{li2024general,
  author = {Yinheng Li and Junhao Wang and Yijie Cao},
  title = {{A General Framework on Enhancing Portfolio Management with Reinforcement Learning}},
  booktitle = {Iccmb},
  year = {2024},
  doi = {10.1145/3647782.3647801},
}
Figure