Optimizing Trading Strategies in Quantitative Markets Using Multi-Agent Reinforcement Learning

Ercan Engin Kuruoglu, PhD Cantab, Prof, Ieee Fellow, Hengxi Zhang, Zhendong Shi, Yuanquan Hu, Wenbo Ding, Ercan E. Kuruoglu, Xiao-Ping Zhang

TL;DR

This paper introduces two novel multi-agent reinforcement learning (MARL) strategies, CPPI-MADDPG and TIPP-MADDPG, that combine established portfolio insurance techniques with a state-of-the-art MARL framework. These methods demonstrate superior performance in quantitative trading compared to traditional approaches and even a baseline MARL strategy, offering a more robust and adaptive way to navigate volatile markets.

What Problem Does This Paper Address?

Quantitative markets are notoriously complex, characterized by rapid changes and inherent uncertainties. Traditional trading methods struggle to keep pace, and human traders are susceptible to emotional biases leading to irrational decisions. While reinforcement learning (RL) offers a promising avenue for optimal control in dynamic environments, existing RL approaches often focus on single agents. This paper addresses the challenge of developing effective trading strategies in these complex markets by leveraging the power of multi-agent reinforcement learning (MARL) to manage portfolios with diverse strategies and adapt to market dynamics.

Methods

The core of this work lies in integrating two established portfolio insurance strategies, Constant Proportion Portfolio Insurance (CPPI) and Time-Invariant Portfolio Protection (TIPP), with the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework.

  • MADDPG: This is a multi-agent actor-critic method designed for continuous action spaces. Each agent has its own actor network (policy) and critic network (value function). The key innovation here is its application to a multi-agent trading scenario.
  • CPPI: This strategy maintains a portfolio by allocating assets between a "risky asset" and a "protection floor." The risky asset allocation is proportional to the "cushion" (total asset minus the floor), with a risk factor determining aggressiveness.
  • TIPP: A variation of CPPI, TIPP adjusts the protection floor dynamically based on the total asset value and the previous floor, offering a more adaptive safety net.
  • Integration: The paper proposes CPPI-MADDPG and TIPP-MADDPG, where the output actions from the MADDPG agents are constrained or adjusted by the CPPI or TIPP rules. This allows the MARL agents to learn optimal policies while adhering to risk management principles. The loss function for each agent includes a term to encourage correlation between agents, promoting portfolio diversity.

Datasets

The experiments were conducted using real-world stock market data from the Shenzhen Stock Exchange.

  • Training Data: January 1st, 2018, to December 31st, 2020.
  • Testing Data: January 1st, 2021, to December 31st, 2021.
  • Securities: 100 stocks from the Shenzhen Stock Exchange, plus cash as a risk-free asset.
  • Benchmark: The paper utilizes the FinRL [20] environment, which provides a simulated quantitative trading environment.

Key Insights and Ideas

  • MARL for Portfolio Insurance: Combining MARL with established portfolio insurance strategies like CPPI and TIPP is a powerful approach for quantitative trading. It allows for both adaptive learning and inherent risk management.
  • Adaptive Risk Management: TIPP's dynamic floor adjustment offers a significant advantage over CPPI's fixed floor, particularly in mitigating "gap risk" where sudden market drops can breach the floor value.
  • Agent Cooperation/Diversity: The inclusion of a correlation term in the loss function encourages agents to adopt diverse strategies, leading to a more robust and diversified portfolio.
  • Tailoring to Investor Preferences: The proposed methods can be tuned via parameters (like the risk factor k and floor percentage ε) to align with individual investor risk appetites.

Key Results

  • Superior Performance: Both CPPI-MADDPG and TIPP-MADDPG consistently outperformed traditional counterparts and a baseline MADDPG strategy in quantitative trading.
  • Annual Return (AR): TIPP-MADDPG achieved the highest AR (9.68%), followed by CPPI-MADDPG (7.76%) and MADDPG (8.22%).
  • Sharpe Ratio (SR): CPPI-MADDPG achieved the highest SR (2.18), indicating better risk-adjusted returns.
  • Maximum Drawdown (MaxD): CPPI-MADDPG demonstrated the lowest MaxD (6.60%), signifying better protection against portfolio value drops.
  • Qualitative Analysis: Visualizations (Fig. 3) showed that MADDPG agents tended towards uniform asset allocation, while CPPI-MADDPG and TIPP-MADDPG exhibited sparser allocations, suggesting more strategic and selective investment.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the proposed strategies can revert to the baseline MADDPG under certain parameter settings, implying that the insurance strategies are not always strictly enforced or might be overridden by the learned policy.
  • The performance of Universal Portfolio (UP) was noted to struggle with real-time data in their test set.
  • MADQN's limitation in exploration capacity made it less adept in uncertain markets.
  • Future Work: The authors express optimism about the potential for broader applications and adaptations of their methodologies in quantitative markets, given the promising results and the evolving nature of these markets. Further research could explore more complex MARL architectures, different insurance strategies, or more sophisticated market simulators.

Summary updated

BibTeX
@inproceedings{fellow2024optimizing,
  author = {Ercan Engin Kuruoglu, PhD Cantab, Prof, IEEE Fellow and Hengxi Zhang and Zhendong Shi and Yuanquan Hu and Wenbo Ding and Ercan E. Kuruoglu and Xiao-Ping Zhang},
  title = {{Optimizing Trading Strategies in Quantitative Markets Using Multi-Agent Reinforcement Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446489},
}
Figure