Strategic Trading in Quantitative Markets through Multi-Agent Reinforcement Learning

Ercan Engin Kuruoglu, PhD Cantab, Prof, Ieee Fellow, Hengxi Zhang, Zhendong Shi, Yuanquan Hu, Wenbo Ding, Ercan E. Kuruoglu, Xiao-Ping Zhang

Introduction

Quantitative trading in modern financial markets is characterized by high volatility and "noisy" signals stemming from stochastic behaviors and unpredictable global events. While algorithmic trading has become standard, traditional models often struggle with the dynamic nature of these markets. Single-agent Reinforcement Learning (RL) has emerged as a potential solution, yet it frequently lacks the necessary risk-distribution capabilities and diversity required for complex portfolio management. This research addresses these challenges by introducing a hybrid framework that fuses Multi-Agent Reinforcement Learning (MARL) with established portfolio insurance strategies to maximize returns while ensuring strict downside protection.

Methodology

Hybrid Framework: Integrating MARL with Portfolio Insurance

The core of the proposed system is the integration of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm with rule-based financial strategies. MADDPG utilizes a decentralized actor-centralized critic architecture, allowing multiple agents to learn coordinated trading policies.

To prevent the "hallucinatory" risks often associated with pure RL, the researchers implemented Constant Proportion Portfolio Insurance (CPPI) and Time-Invariant Portfolio Protection (TIPP) as mathematical constraints on agent actions.

The CPPI strategy mechanism: allocating a multiple of the 'cushion' (Total Asset - Protection Floor) into risky assets.

As shown in Figure 1, CPPI maintains a "protection floor" ($F$). The difference between the total assets ($A$) and this floor is the "cushion" ($C$). The strategy invests a multiple ($k$) of this cushion into risky assets ($E = k * C$). TIPP extends this by allowing the protection floor to rise as portfolio value increases, providing a more dynamic safety net.

Diversity and Risk Management

To mitigate "multi-agent collapse"—where agents converge on identical, sub-optimal strategies—a correlation penalty was added to the loss function. This encourages agents to explore diverse asset allocations, effectively distributing risk across the portfolio.

Experimental Setup

The models were validated using a dataset of 100 stocks from the Shenzhen Stock Exchange (via Tushare). The training period spanned from 2018 to 2020, with testing conducted throughout 2021. The environment, built on the FinRL framework, incorporated realistic constraints such as transaction costs and non-negative balance requirements.

Performance Evaluation

The hybrid models demonstrated significant improvements over traditional benchmarks and standard MARL approaches.

Strategy Annual Return (AR) Sharpe Ratio (SR) Max Drawdown (MaxD)
Universal Portfolio (UP) 3.36% 1.03 3.86%
MADDPG (Standard) 8.22% 1.99 12.26%
CPPI-MADDPG 7.76% 2.18 6.60%
TIPP-MADDPG 9.68% 2.09 9.02%
Cumulative total asset performance of the proposed TIPP-MADDPG and CPPI-MADDPG models against baselines.

As illustrated in Figure 3, TIPP-MADDPG achieved the highest cumulative returns, while CPPI-MADDPG provided the best risk-adjusted performance, evidenced by its superior Sharpe Ratio and significantly reduced maximum drawdown compared to standard MADDPG.

Key Insights

  • Safety Rails: Combining RL adaptability with financial theory constraints prevents excessive risk-taking.
  • Sparse Allocation: Visualization of agent behavior revealed that CPPI/TIPP-augmented agents learned "sparse" allocation patterns, focusing on high-confidence opportunities rather than uniform, inefficient distribution.
  • Resilience: TIPP’s dynamic floor adjustment proved more resilient to "gap risk"—the danger of rapid asset devaluation—than the static floor of CPPI.

Conclusion and Future Work

The fusion of MARL with portfolio insurance strategies offers a robust path forward for automated quantitative trading. While successful, future research should address the sensitivity of parameters like the risk factor ($k$) and floor percentage ($\phi$). Additionally, expanding the scope to include diverse asset classes (e.g., cryptocurrencies) and scaling the MARL architecture to handle larger numbers of agents remain key areas for development.

Summary updated

BibTeX
@inproceedings{fellow2024strategic,
  author = {Ercan Engin Kuruoglu, PhD Cantab, Prof, IEEE Fellow and Hengxi Zhang and Zhendong Shi and Yuanquan Hu and Wenbo Ding and Ercan E. Kuruoglu and Xiao-Ping Zhang},
  title = {{Strategic Trading in Quantitative Markets through Multi-Agent Reinforcement Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446489},
}
Figure