Combining transformer based deep reinforcement learning with Black-Litterman model for portfolio optimization

Ruoyu Sun, Angelos Stefanidis, Zhengyong Jiang, Jionglong Su

Abstract

Portfolio optimization remains a cornerstone of financial engineering, yet traditional methods often suffer from estimation errors, while modern deep learning approaches frequently fail to capture complex asset correlations. This paper introduces the Black-Litterman model-based Deep Reinforcement Learning Agent (BDA), a hybrid framework that integrates the predictive capabilities of Transformers with the statistical stability of the Black-Litterman (BL) model. By generating subjective views through a modified Transformer architecture and refining them via a Bayesian optimizer, the BDA achieves superior risk-adjusted returns. Empirical evaluations on the Dow Jones Industrial Average (DJIA) demonstrate that the BDA outperforms state-of-the-art strategies by over 42% in accumulated returns.

1. Introduction

Modern portfolio management is defined by the tension between risk and return. Since Harry Markowitz introduced mean-variance theory, the industry has relied on mathematical frameworks to balance these factors. However, the classical approach is notoriously sensitive to input parameters; small errors in expected return estimates can lead to extreme, unstable asset allocations—a phenomenon known as "error maximization."

Deep Reinforcement Learning (DRL) has emerged as a potential solution, capable of adapting to non-linear market dynamics. Yet, standard DRL agents often struggle with high-dimensional financial data and fail to learn the dynamic correlations between assets necessary for effective hedging. Furthermore, the common use of a softmax layer for weight allocation lacks the rigorous mathematical grounding required for high-stakes environments. This research proposes the BDA framework to bridge the gap between AI-driven prediction and classical statistical optimization.

2. Methodology

2.1 The BDA Framework

The BDA agent is structured into three distinct modules designed to process market data, generate insights, and optimize allocations:

  1. Information Extraction: The system processes historical price fluctuations as tensors, providing a multi-dimensional view of market history.
  2. Decision-Making (The AI Engine):
    • Modified Transformer: To capture asset correlations, the authors utilized a Transformer architecture but intentionally removed the position encoding module. In natural language processing, position encoding tracks the order of words; in finance, removing it forces the model to focus on the concurrent relationships between assets rather than purely sequential patterns, thereby reducing overfitting to specific time-steps.
  3. Risk Aversion CNN: A Convolutional Neural Network (CNN) evaluates the current market state to determine the agent's risk aversion level, allowing the model to become more conservative during periods of high volatility.
  4. Portfolio Optimizer (The Statistical Anchor): The outputs from the Transformer (subjective views) and the CNN (risk aversion) are fed into the Black-Litterman model. The BL model acts as a Bayesian bridge, combining these AI-driven views with historical market equilibrium to produce stable, optimized portfolio weights.

2.2 Training Strategy

To address the instability often found in DRL training (such as exploding gradients), the researchers implemented a differentiable objective function and a "target value" training strategy. Rather than a simple reward maximization approach, this strategy uses the target value as a regularizer, ensuring the model converges reliably and generalizes better to unseen market conditions.

3. Experimental Setup

The model was evaluated using a dataset derived from the Dow Jones Industrial Average (DJIA), covering 29 constituent stocks from January 2018 to December 2022. The researchers conducted four overlapping experiments, utilizing a three-year training window followed by a 120-trading-day back-test. This setup tested the model's ability to handle various market cycles, including the volatility of the early 2020s.

4. Results and Discussion

4.1 Performance Metrics

The BDA agent demonstrated significant dominance over traditional and machine learning benchmarks, including Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Online Newton Step (ONS):

  • Accumulated Returns: The BDA achieved at least 42% higher accumulated returns than the best-performing comparative strategy.
  • Risk-Adjusted Performance: The model consistently secured the highest Sharpe and Sortino Ratios, indicating that its returns were not merely the result of higher risk-taking but of superior asset selection and timing.
  • Ablation Studies: To verify the contribution of each component, the authors tested versions of the model without the BL optimizer or the modified Transformer. These versions saw performance drops of at least 24%, confirming that the synergy between the Transformer's correlation learning and the BL model's stability is critical to the system's success.

4.2 Key Insights

The success of the BDA highlights two major findings. First, the removal of position encoding in Transformers is a viable technique for financial time-series where cross-asset correlation is more informative than absolute temporal positioning. Second, the Black-Litterman model serves as an effective "safety rail," preventing the DRL agent from making erratic allocations based on noisy AI predictions.

5. Conclusion and Future Work

The BDA framework represents a significant step forward in hybrid financial modeling. By combining the strengths of Transformers and Bayesian optimization, it overcomes the "error maximization" of classical models and the correlation blindness of standard DRL.

Future research will focus on relaxing the Gaussian distribution assumptions inherent in the Black-Litterman model by exploring Elliptical distributions to better account for market "fat tails." Additionally, the team aims to investigate multi-agent hierarchical reinforcement learning to manage more complex, multi-sector portfolios.

Summary updated

BibTeX
@article{sun2024combining,
  author = {Ruoyu Sun and Angelos Stefanidis and Zhengyong Jiang and Jionglong Su},
  title = {{Combining transformer based deep reinforcement learning with Black-Litterman model for portfolio optimization}},
  journal = {Neural Computing and Applications},
  year = {2024},
  doi = {10.1007/s00521-024-09805-9},
}
Figure