Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning

Yifan Zhang, Peilin Zhao, Qingyao Wu, Bin Li, Junzhou Huang, Mingkui Tan

TL;DR

This paper introduces the Portfolio Policy Network (PPN), a deep reinforcement learning framework that automates asset allocation while explicitly accounting for transaction costs and market risk. By using a novel two-stream architecture to capture both individual asset trends and cross-asset correlations, PPN achieves significantly higher risk-adjusted returns than state-of-the-art baselines on real-world cryptocurrency and stock datasets.

What Problem Does This Paper Address?

Portfolio selection—the act of dynamically reallocating wealth across various assets—is a cornerstone of finance. However, two major hurdles make it a "wicked" problem for AI:

  1. Complex Data Representation: Financial price series are non-stationary, noisy, and full of "jumps." Furthermore, assets don't move in isolation; their correlations (e.g., how Ethereum moves when Bitcoin drops) are crucial but difficult to model simultaneously with time-series patterns.
  2. The "Practicality Gap": Many theoretical models ignore transaction costs (commissions and taxes) and risk costs (volatility). A model that ignores transaction costs might trade too aggressively, "bleeding out" wealth through fees. A model that ignores risk might chase high returns into a market crash.

The authors aim to bridge this gap by creating a model that "sees" the whole market and "feels" the cost of every trade.

Methods

The authors propose the Portfolio Policy Network (PPN), which stands out due to two primary innovations:

1. Two-Stream Feature Extraction

Instead of a single neural network, PPN uses two specialized streams:

  • Sequential Information Net: Utilizes LSTMs (Long Short-Term Memory units) to extract temporal patterns from individual asset price histories.
  • Correlation Information Net: Introduces a novel Temporal Correlational Convolution Block (TCCB). This uses dilated causal convolutions to expand the "receptive field" (looking further back in time without leaking future data) and correlational convolutions to fuse features across different assets at every time step.

2. Cost-Sensitive Reward Function

Standard Reinforcement Learning (RL) often optimizes for simple cumulative returns. PPN uses a sophisticated reward function that includes:

  • Log-returns: The standard measure of growth.
  • Risk Penalty: A variance-based penalty that discourages highly volatile strategies.
  • Transaction Cost Constraint: A penalty based on the $L_1$ norm of the change in portfolio weights, discouraging unnecessary or excessive trading.

The model is trained using a Direct Policy Gradient (DPG) algorithm, which the authors theoretically prove can approach the "log-optimal" growth rate even in the presence of costs.

Datasets

The model was evaluated on two distinct types of real-world data:

  • Cryptocurrency (Poloniex): Four datasets (Crypto-A to D) containing between 12 and 44 assets. These are high-volatility environments with 30-minute trading intervals.
  • Stock Market (S&P 500): A dataset of 506 stocks to test the model's ability to generalize to traditional, lower-volatility financial markets.
  • Features: The input includes four price types: Open, High, Low, and Close.

Key Insights and Ideas

  • Correlations are a "Force Multiplier": The authors found that models considering asset correlations (PPN) significantly outperformed those that treated assets independently (PPN-I). Understanding how assets move together allows for better hedging and diversification.
  • The Recursive Advantage: PPN takes the previous portfolio state as an input. This "memory" of current holdings is what allows the RL agent to realize that a small predicted gain might not be worth the transaction cost required to rebalance the portfolio.
  • Robustness to High Fees: In experiments where transaction costs were artificially inflated (up to 5%), PPN learned a "passive" strategy—essentially holding its position to avoid fees—while other state-of-the-art models continued to trade and lost nearly all their wealth.

Key Results

The PPN model demonstrated clear superiority across multiple metrics:

  • Profitability: On the Crypto-A dataset, PPN achieved an Accumulated Portfolio Value (APV) of 32.04, dwarfing the previous state-of-the-art (EIIE) which scored 10.48.
  • Risk-Adjusted Return: PPN consistently achieved the highest Sharpe Ratio and Calmar Ratio, indicating that its profits weren't just the result of taking higher risks, but of smarter allocation.
  • Generalization: On the S&P 500 dataset, PPN achieved an APV of 167.84, significantly higher than EIIE (99.35) and traditional strategies like Uniform Buy-And-Hold (1.21).

Limitations and Future Work

  • Market Impact: The study assumes "zero-market-impact," meaning the agent's trades are small enough that they don't move the market price. In reality, large institutional trades cause "slippage," which isn't modeled here.
  • Data Scope: The model relies purely on price data (technical analysis). The authors suggest that incorporating social text information (sentiment analysis from news or Twitter) could further improve the model's ability to anticipate "black swan" events or sudden market shifts.
  • Liquidity: The model assumes perfect liquidity (you can always buy/sell at the current price), which may not hold for smaller "altcoins" in the crypto space.

Summary updated

Abstract

Portfolio Selection is an important real-world financial task and has attracted extensive attention in artificial intelligence communities. This task, however, has two main difficulties: (i) the non-stationary price series and complex asset correlations make the learning of feature representation very hard; (ii) the practicality principle in financial markets requires controlling both transaction and risk costs. Most existing methods adopt handcraft features and/or consider no constraints for the costs, which may make them perform unsatisfactorily and fail to control both costs in practice. In this paper, we propose a cost-sensitive portfolio selection method with deep reinforcement learning. Specifically, a novel two-stream portfolio policy network is devised to extract both price series patterns and asset correlations, while a new cost-sensitive reward function is developed to maximize the accumulated return and constrain both costs via reinforcement learning. We theoretically analyze the near-optimality of the proposed reward, which shows that the growth rate of the policy regarding this reward function can approach the theoretical optimum. We also empirically evaluate the proposed method on real-world datasets. Promising results demonstrate the effectiveness and superiority of the proposed method in terms of profitability, cost-sensitivity and representation abilities.
BibTeX
@article{zhang2022cost,
  author = {Yifan Zhang and Peilin Zhao and Qingyao Wu and Bin Li and Junzhou Huang and Mingkui Tan},
  title = {{Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2022},
  url = {https://openreview.net/forum?id=xFo-zzI_8Hk},
}

Loading related content...

Figure