Abstract
Generating an investment strategy using advanced deep learning methods in stock markets has recently been a topic of interest. Most existing deep learning methods focus on proposing an optimal model or network architecture by maximizing return. However, these models often fail to consider and adapt to the continuously changing market conditions. In this paper, we propose the Multi-Agent reinforcement learning-based Portfolio management System (MAPS). MAPS is a cooperative system in which each agent is an independent "investor" creating its own portfolio. In the training procedure, each agent is guided to act as diversely as possible while maximizing its own return with a carefully designed loss function. As a result, MAPS as a system ends up with a diversified portfolio. Experiment results with 12 years of US market data show that MAPS outperforms most of the baselines in terms of Sharpe ratio. Furthermore, our results show that adding more agents to our system would allow us to get a higher Sharpe ratio by lowering risk with a more diversified portfolio.
Introduction
Most deep learning models in finance focus on a single goal: maximizing total return. However, in the real world, professional investors prioritize risk-adjusted returns. A model that achieves a 20% return with high volatility is often less desirable than one that achieves 15% with minimal fluctuations.
Existing single-agent reinforcement learning (RL) models often struggle to adapt to shifting market conditions and fail to diversify their strategies. Traditional quantitative strategies, such as momentum or mean reversion, only capture narrow slices of market dynamics. MAPS addresses this by positing that a robust system requires a variety of independent strategies working in concert to mitigate risk, rather than a single "optimal" strategy.
Proposed Methodology
MAPS is a cooperative Multi-Agent Reinforcement Learning framework designed to treat portfolio management as a collaborative game.
System Architecture
The system consists of $K$ agents, where each agent is a neural network comprising an MLP encoder and a Q-network. All agents receive the same market state—historical closing prices—as input. Each agent independently determines whether to take a "Long," "Short," or "Neutral" position for every stock in the universe. These individual decisions are then aggregated into a final, diversified "portfolio of portfolios."
Strategic Diversification via Global Loss
The primary innovation of MAPS is the introduction of a Global Loss term. Standard RL agents tend to converge on the same "best" strategy, which eliminates the benefits of a multi-agent setup. The Global Loss function calculates the correlation between agents' positional confidence scores. If two agents begin making identical bets, the system penalizes them. This forces agents to explore uncorrelated ways to generate alpha, mimicking a large investment firm with multiple desks running distinct strategies.
Experimental Setup
The authors evaluated MAPS using 18 years of daily closing price data (2000–2018) for approximately 3,000 US companies listed in the Russell 3000 index. The data was partitioned into three phases:
- Training: 2000–2004
- Validation: 2004–2006
- Testing: 2006–2018 (further split to demonstrate performance across different market cycles).
Results and Discussion
MAPS was compared against traditional strategies (Momentum, Mean Reversion) and state-of-the-art deep learning models, including CNNs, LSTMs, and DA-RNNs.
Performance Metrics
In the 2006–2012 test period, MAPS@16 achieved a Sharpe ratio of 5.547, significantly outperforming the next best model, DA-RNN (3.283), and traditional Momentum (1.149). While other models, such as CNNs, experienced negative returns during certain volatile periods, MAPS maintained positive annualized returns across both test windows.
Key Insights
- Strategy-Level Diversification: By mathematically penalizing agents for "thinking alike," the model achieves diversification at the strategy level, not just the asset level.
- Scaling with Agents: As the number of agents increases (e.g., from MAPS@4 to MAPS@16), the Sharpe ratio generally improves. This suggests that the "disagreements" between agents effectively cancel out idiosyncratic risks.
- Emergent Hedging: Case studies of volatile periods (such as Amazon in 2014) revealed that some agents naturally took long positions while others went short. This internal hedging allows the system to survive market corrections that often wipe out single-strategy models.
Conclusion and Future Work
MAPS demonstrates that multi-agent cooperation, enforced by a diversity-promoting loss function, can significantly enhance risk-adjusted returns in portfolio management. Future work may involve refining the learning scheme for larger agent populations, as current results show non-linear scaling in correlation reduction. Additionally, incorporating fundamental data, technical indicators, or sentiment analysis could further enhance the agents' ability to discover unique market patterns.
Summary updated