Abstract
This paper introduces DPS-RRL (Direct Portfolio Selection using Recurrent Reinforcement Learning), a novel framework for end-to-end portfolio management. Unlike traditional methods that aggregate individual asset signals heuristically, DPS-RRL learns to allocate investment weights across multiple assets simultaneously by directly optimizing the portfolio's Sharpe ratio. Experimental results on synthetic datasets demonstrate that this "top-down" approach significantly improves risk-adjusted returns compared to standard benchmarks.
Introduction
In algorithmic trading, Recurrent Reinforcement Learning (RRL) has traditionally been applied to single-asset trading systems. When managing a portfolio, researchers typically employ a "bottom-up" strategy: training separate RRL agents for each stock and then applying a heuristic—such as equal weighting—to combine their signals. This fragmented approach fails to account for the mathematical interdependencies between assets during the learning phase, creating a performance bottleneck.
To address this, we propose a "top-down" approach where the model learns optimal allocation weights directly. By treating the entire portfolio as a single differentiable system, the model can internalize the relationships between assets and optimize for global performance metrics.
Methodology
The DPS-RRL framework is built on four key pillars:
- Direct Weight Allocation: The model outputs a vector of weights ($F_t$) for all $m$ securities. These weights are processed through a softmax-like function to ensure a "long-only" portfolio (non-negative weights summing to one).
- Recurrent Features: Input features include a look-back window of historical returns and the portfolio weights from the previous period ($F_{t-1}$). This recurrence allows the model to maintain state and account for the current portfolio composition.
- Differentiable Objective: The system is trained to maximize the Sharpe Ratio, a standard measure of return per unit of risk. Since the profit function (including transaction costs) and the Sharpe Ratio formula are differentiable, we utilize Gradient Ascent to update system parameters directly.
- Transaction Cost Awareness: The model explicitly incorporates a transaction cost rate ($\delta$) into its profit calculations. By including these costs in the gradient, the model learns to avoid excessive rebalancing that would diminish net returns.
Experiments and Results
We validated the DPS-RRL framework using an artificial dataset composed of three price series representing different securities, generated via random walks with autoregressive trend processes. The data was split equally into training and out-of-sample testing sets.
In out-of-sample testing, DPS-RRL significantly outperformed all standard benchmarks, including those given the advantage of zero transaction costs.
As shown in the results, the DPS-RRL strategy achieved a Sharpe ratio of 0.1723, nearly four times higher than the best-performing benchmark (Equally Weighted Buy & Hold). Furthermore, it generated over five times the cumulative profit of the nearest competitor, demonstrating the effectiveness of direct weight optimization.
Discussion and Conclusion
The core insight of this research is that portfolio selection should not be decoupled from asset-trading logic. By optimizing weights directly against the Sharpe ratio, the model learns to balance assets dynamically while adhering to real-world constraints like transaction costs.
While these results are promising, the study is currently limited by its reliance on synthetic data, which may not fully capture the non-stationarity and "fat-tail" risks of real-world markets. Future work will focus on testing the DPS-RRL method on broader, real-world market datasets and incorporating re-investment strategies to further align the model with practical portfolio management standards.
Summary updated