MAPS: Multi-Agent reinforcement learning-based Portfolio management System

Jinho Lee, Raehyun Kim, Seok-Won Yi, Jaewoo Kang

Abstract

Generating an investment strategy using advanced deep learning methods in stock markets has recently been a topic of interest. Most existing deep learning methods focus on proposing an optimal model or network architecture by maximizing return. However, these models often fail to consider and adapt to the continuously changing market conditions. In this paper, we propose the Multi-Agent reinforcement learning-based Portfolio management System (MAPS). MAPS is a cooperative system in which each agent is an independent "investor" creating its own portfolio. In the training procedure, each agent is guided to act as diversely as possible while maximizing its own return with a carefully designed loss function. As a result, MAPS as a system ends up with a diversified portfolio. Experiment results with 12 years of US market data show that MAPS outperforms most of the baselines in terms of Sharpe ratio. Furthermore, our results show that adding more agents to our system would allow us to get a higher Sharpe ratio by lowering risk with a more diversified portfolio.

Introduction

In quantitative trading, most deep learning models are "return-hungry," designed primarily to maximize raw profit. However, professional portfolio management requires a delicate balance between return and risk. A model yielding 20% profit with high volatility is often less desirable than one yielding 15% with steady growth.

MAPS (Multi-agent Reinforcement Learning-based Portfolio Management System) addresses this by treating portfolio management as a cooperative team sport. By training a group of independent "investor" agents to maximize their own returns while being explicitly penalized for acting too much like their peers, the system creates a highly diversified "portfolio of portfolios" that significantly outperforms traditional models in risk-adjusted returns (Sharpe ratio).

Methodology

MAPS uses a cooperative Multi-Agent Reinforcement Learning (MARL) framework where each agent is a Deep Q-Network (DQN). To overcome standard pitfalls like non-stationarity and the curse of dimensionality, the system employs several key innovations:

  • Structural Diversity: Instead of identical agents, each agent possesses a slightly different neural network depth and width, encouraging them to learn distinct patterns from identical data.
  • The Global Loss Function: This is the core innovation. The total loss for each agent is a weighted sum of a Local Loss (standard reward maximization) and a Global Loss. The Global Loss calculates the correlation between an agent's positional confidence and its peers. If an agent's trades become too similar to the group, it is penalized.
  • Portfolio of Portfolios: All agents observe the same market window (historical prices) but produce independent weights. The final system output is an ensemble that sums the suggested portfolios of all $K$ agents to create a final, diversified asset allocation.
The MAPS architecture utilizes a shared state input and a dual loss function (Local and Global) to encourage agent diversity.

Experimental Setup

The researchers evaluated MAPS using a massive dataset of US market data comprising approximately 3,000 companies from the Russell 3000 index. The timeframe spans 18 years of daily closing prices (2000–2018), split into training (2000–2004), validation (2004–2006), and two testing periods (2006–2012 and 2012–2018) to ensure robustness across different market cycles.

Key Results and Analysis

MAPS was compared against traditional strategies (Momentum, Mean-Reversion) and state-of-the-art deep learning models (CNNs, LSTMs, and the DA-RNN).

Cumulative asset value of MAPS variants compared to traditional and deep learning baselines from 2006 to 2018.
  • Superior Risk-Adjusted Return: In the 2006–2012 period, MAPS@16 (16 agents) achieved a Sharpe ratio of 5.547, dwarfing the next best baseline (DA-RNN at 3.283).
  • Consistency Across Cycles: While models like the CNN lost money in the 2012–2018 period, MAPS maintained positive returns (~5.5%) and a strong Sharpe ratio (>2.2).
  • The "Committee" Effect: Case studies show that during volatile periods, agents often take opposing positions (some "Long," others "Short"). This internal disagreement prevents the system from over-leveraging on a single, uncertain direction, effectively smoothing the equity curve.

Conclusion

MAPS demonstrates that forced individuality through a Global Loss function allows a multi-agent system to find "niche" ways to be profitable, mimicking a room of human traders with different specialties. While adding more agents generally improves the Sharpe ratio by reducing idiosyncratic risk, future work will focus on incorporating alternative data (like news sentiment) and dynamic tuning of the Global Loss weighting to further optimize performance.

Summary updated

Abstract

Generating an investment strategy using advanced deep learning methods in stock markets has recently been a topic of interest. Most existing deep learning methods focus on proposing an optimal model or network architecture by maximizing return. However, these models often fail to consider and adapt to the continuously changing market conditions. In this paper, we propose the Multi-Agent reinforcement learning-based Portfolio management System (MAPS). MAPS is a cooperative system in which each agent is an independent "investor" creating its own portfolio. In the training procedure, each agent is guided to act as diversely as possible while maximizing its own return with a carefully designed loss function. As a result, MAPS as a system ends up with a diversified portfolio. Experiment results with 12 years of US market data show that MAPS outperforms most of the baselines in terms of Sharpe ratio. Furthermore, our results show that adding more agents to our system would allow us to get a higher Sharpe ratio by lowering risk with a more diversified portfolio.
BibTeX
@inproceedings{lee2020maps,
  author = {Jinho Lee and Raehyun Kim and Seok-Won Yi and Jaewoo Kang},
  title = {{MAPS: Multi-Agent reinforcement learning-based Portfolio management System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=AribRSDjwc},
}
Figure