Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning

Yifan Zhang, Peilin Zhao, Qingyao Wu, Bin Li, Junzhou Huang, Mingkui Tan

Abstract

Portfolio Selection is an important real-world financial task and has attracted extensive attention in artificial intelligence communities. This task, however, has two main difficulties: (i) the non-stationary price series and complex asset correlations make the learning of feature representation very hard; (ii) the practicality principle in financial markets requires controlling both transaction and risk costs. Most existing methods adopt handcraft features and/or consider no constraints for the costs, which may make them perform unsatisfactorily and fail to control both costs in practice. In this paper, we propose a cost-sensitive portfolio selection method with deep reinforcement learning. Specifically, a novel two-stream portfolio policy network is devised to extract both price series patterns and asset correlations, while a new cost-sensitive reward function is developed to maximize the accumulated return and constrain both costs via reinforcement learning. We theoretically analyze the near-optimality of the proposed reward, which shows that the growth rate of the policy regarding this reward function can approach the theoretical optimum. We also empirically evaluate the proposed method on real-world datasets. Promising results demonstrate the effectiveness and superiority of the proposed method in terms of profitability, cost-sensitivity and representation abilities.

Introduction

Portfolio selection—the dynamic reallocation of wealth across various assets—is a cornerstone of modern finance. However, it presents a "wicked" problem for artificial intelligence due to two primary hurdles. First, financial price series are non-stationary and noisy, and assets exhibit complex correlations that are difficult to model simultaneously with temporal trends. Second, real-world trading is constrained by a "practicality gap": models must account for transaction costs (commissions) and risk costs (volatility). A model that ignores these factors may trade too aggressively, eroding wealth through fees, or chase high returns into excessive risk.

The interaction between the portfolio policy network agent and the financial market environment, highlighting the feedback of previous actions to account for transaction costs.

To bridge this gap, we introduce the Portfolio Policy Network (PPN), a deep reinforcement learning (DRL) framework that "sees" the whole market and "feels" the cost of every trade. As shown in Figure 1, the agent interacts with the market environment by receiving a state $s_t$ and a reward, while crucially considering the action from the last period $a_{t-1}$ to calculate transaction costs.

The Portfolio Policy Network (PPN)

The PPN architecture is designed to overcome the limitations of handcrafted features and independent asset modeling through two primary innovations.

Two-Stream Feature Extraction

Instead of a single neural network, PPN utilizes two specialized streams to process input data (Open, High, Low, and Close prices):

  1. Sequential Information Net: This stream uses LSTMs (Long Short-Term Memory units) to extract temporal patterns from individual asset price histories, capturing the non-stationary nature of price series.
  2. Correlation Information Net: This stream introduces a novel Temporal Correlational Convolution Block (TCCB). It employs dilated causal convolutions to expand the receptive field—allowing the model to look further back in time without data leakage—and correlational convolutions to fuse features across different assets at every time step.
The detailed architecture of the Portfolio Policy Network (PPN), showing the parallel Sequential and Correlation Information Nets feeding into a decision-making module.

Cost-Sensitive Reward Function

Standard RL often optimizes for simple cumulative returns. In contrast, PPN employs a sophisticated reward function that integrates log-returns, a variance-based risk penalty, and a transaction cost constraint based on the $L_1$ norm of weight changes. This ensures the agent learns to avoid unnecessary trading. The model is trained using a Direct Policy Gradient (DPG) algorithm, which we theoretically prove can approach the "log-optimal" growth rate even under cost constraints.

Experimental Evaluation

We evaluated PPN on real-world datasets including cryptocurrency (Poloniex) and the S&P 500. The results demonstrate clear superiority across multiple metrics:

  • Profitability: On the Crypto-A dataset, PPN achieved an Accumulated Portfolio Value (APV) of 32.04, significantly outperforming the previous state-of-the-art (EIIE) at 10.48.
  • Risk Management: PPN consistently achieved the highest Sharpe and Calmar Ratios, indicating that its profits were not merely the result of higher risk-taking but of smarter, cost-aware allocation.
  • Robustness: In stress tests with artificially inflated transaction costs (up to 5%), PPN successfully learned a "passive" strategy to preserve wealth, while competing models continued to trade and lost nearly all value.

Conclusion

By combining a two-stream architecture for feature extraction with a cost-sensitive reward function, PPN effectively addresses the complexities of non-stationary data and asset correlations. While future work should address market impact and liquidity constraints for institutional-scale trading, PPN represents a significant step toward practical, automated portfolio management. Our findings highlight that understanding asset correlations and maintaining a "memory" of current holdings are essential for navigating real-world financial markets.

Summary updated

Abstract

Portfolio Selection is an important real-world financial task and has attracted extensive attention in artificial intelligence communities. This task, however, has two main difficulties: (i) the non-stationary price series and complex asset correlations make the learning of feature representation very hard; (ii) the practicality principle in financial markets requires controlling both transaction and risk costs. Most existing methods adopt handcraft features and/or consider no constraints for the costs, which may make them perform unsatisfactorily and fail to control both costs in practice. In this paper, we propose a cost-sensitive portfolio selection method with deep reinforcement learning. Specifically, a novel two-stream portfolio policy network is devised to extract both price series patterns and asset correlations, while a new cost-sensitive reward function is developed to maximize the accumulated return and constrain both costs via reinforcement learning. We theoretically analyze the near-optimality of the proposed reward, which shows that the growth rate of the policy regarding this reward function can approach the theoretical optimum. We also empirically evaluate the proposed method on real-world datasets. Promising results demonstrate the effectiveness and superiority of the proposed method in terms of profitability, cost-sensitivity and representation abilities.
BibTeX
@article{zhang2022cost,
  author = {Yifan Zhang and Peilin Zhao and Qingyao Wu and Bin Li and Junzhou Huang and Mingkui Tan},
  title = {{Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2022},
  url = {https://openreview.net/forum?id=xFo-zzI_8Hk},
}
Figure