From Bandits Model to Deep Deterministic Policy Gradient, Reinforcement Learning with Contextual Information

Ercan Engin Kuruoglu, PhD Cantab, Prof, Ieee Fellow, Zhendong Shi, Xiaoli Wei, Ercan E. KuruogluarXiv 2023 · DOI · arXiv

Abstract

Quantitative trading faces significant challenges due to noisy signals and non-Gaussian reward distributions characterized by "fat tails." This paper presents a roadmap for evolving Multi-Armed Bandit (MAB) models into complex Deep Deterministic Policy Gradient (DDPG) agents. By incorporating asymmetric $\alpha$-stable distributions to handle market risks and integrating the Constant Proportion Portfolio Insurance (CPPI) strategy into the RL loss function, the authors demonstrate accelerated learning and superior risk-adjusted returns. Experimental results show that while bandit models can achieve high raw returns, the CPPI-DDPG approach offers exceptional stability and capital preservation.

Introduction

Quantitative trading is a high-stakes environment where extreme volatility and "black swan" events are more common than standard models predict. Traditional Reinforcement Learning (RL) agents often struggle in this domain because they are data-hungry, slow to converge, and frequently assume Gaussian noise. This research addresses the critical gap of effectively utilizing contextual information—such as P/E ratios and market trends—to make agents robust enough for real-world financial markets where rewards are often skewed and heavy-tailed.

Methodology

The paper proposes a progression of algorithms that bridge the gap between exploration-focused bandits and control-focused deep RL agents.

Evolutionary roadmap from Multi-armed Bandits to Deep Deterministic Policy Gradient (DDPG).

1. Asymmetric $\alpha$-Thompson Sampling

Unlike standard Thompson Sampling that assumes Gaussian rewards, this method uses $\alpha$-stable distributions. These are mathematically designed to model systems with heavy tails and skewness, making them ideal for financial shocks. As illustrated in the relationship between stochastic processes, the "Stable" and "Levy" processes provide a more accurate foundation for modeling market volatility than simple Gaussian assumptions.

Relationship between various stochastic processes, highlighting the Stable and Levy processes used for modeling market volatility.

2. Contextual and Semi-Contextual Bandits

The authors extend the bandit framework to include feature vectors (context). They introduce "Semi-Contextual" versions that perform local updates for specific scenarios, reducing the time spent exploring irrelevant state-action pairs.

3. CPPI-DDPG and Novel Loss Function

The "heavy lifter" of this research is the modification of DDPG—an actor-critic algorithm for continuous actions. The authors integrate Constant Proportion Portfolio Insurance (CPPI), a classic financial strategy that maintains a "protection floor" for assets, directly into the RL loss function. This acts as a form of "expert supervision," constraining the search space and helping the agent find profitable strategies faster. Additionally, a correlation penalty is included to ensure portfolio diversification among multiple agents.

Experimental Evaluation

The approach was validated across several domains to prove its versatility:

  • Synthetic Data: Generated from $\alpha$-stable distributions to test robustness.
  • Low-Frequency Trading (Tushare): Shenzhen Stock Exchange data with 10 contextual features (e.g., P/E and P/B ratios).
  • High-Frequency Trading (Yahoo Finance): Tick-level data for S&P 500 stocks.
  • Recommendation Systems: The MovieLens 100K dataset was used to demonstrate the algorithm's generalizability.

Key Results and Discussion

The experimental results highlight a distinct trade-off between raw returns and risk management:

  • Returns: The Asymmetric Thompson Sampling (AD-TS) algorithm achieved the highest annual return at 9.68%, compared to 8.22% for standard DDPG.
  • Risk-Adjusted Performance: The CPPI-DDPG model excelled in stability, reaching a Sharpe Ratio of 23.5% and a Maximum Drawdown of only 3.39%. This suggests that while bandit models might find higher peaks, the CPPI-integrated RL model is significantly safer for actual capital preservation.
  • Convergence: Incorporating contextual information and CPPI supervision significantly reduced "regret" during early training stages compared to "blind" RL models.

Conclusion

This research demonstrates that domain knowledge, such as CPPI, can serve as a powerful regularizer for RL agents in finance. While CPPI-DDPG is excellent for short-term stability, standard DDPG may eventually offer higher long-term potential when not constrained by fixed rules. Future work will focus on a fully Bayesian framework for sequential learning, allowing agents to update their internal world models more dynamically as they encounter unprecedented market conditions.

Summary updated

BibTeX
@misc{fellow2023from,
  author = {Ercan Engin Kuruoglu, PhD Cantab, Prof, IEEE Fellow and Zhendong Shi and Xiaoli Wei and Ercan E. Kuruoglu},
  title = {{From Bandits Model to Deep Deterministic Policy Gradient, Reinforcement Learning with Contextual Information}},
  year = {2023},
  eprint = {2310.00642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00642},
}
Figure