Augmented Lagrangian Risk-constrained Reinforcement Learning for Portfolio Optimization (Student Abstract)

Bayaraa Enkhsaikhan, Ohyun JoAAAI 2025 · DOI

Introduction

In high-stakes financial environments, portfolio optimization requires a delicate balance between maximizing returns and strictly managing downside risk. Traditional methods often struggle with the dynamic and volatile nature of real-world markets. While Reinforcement Learning (RL) offers a flexible alternative, standard agents frequently prioritize high rewards at the expense of safety, often violating regulatory or internal risk limits. This research addresses the challenge of enforcing "hard" risk constraints within an RL framework, ensuring that agents adhere to safety requirements like Value-at-Risk (VaR) without significantly compromising performance.

Methodology

The proposed framework builds upon the Deep Deterministic Policy Gradient (DDPG) algorithm, enhanced with a specialized architecture to handle risk constraints. The system utilizes three distinct neural networks:

  1. Actor Network: Determines the optimal asset allocation (weights) for the portfolio.
  2. Critic Network: Estimates the expected return for a given state-action pair.
  3. Cost Network: Acts as a safety monitor, predicting the expected risk (specifically VaR) associated with the actor's decisions.
Architecture of the proposed risk-constrained RL framework featuring Actor, Critic, and Cost networks.

To enforce hard constraints, the researchers employ the Augmented Lagrangian Multiplier (ALM) method. Unlike standard soft penalties, ALM combines a Lagrangian multiplier with a quadratic penalty. This dual approach effectively penalizes the agent more severely as it approaches or crosses risk boundaries, forcing it back into the feasible (safe) region during training.

Furthermore, the model utilizes a Dirichlet distribution for exploration. This is a critical design choice for portfolio management, as it naturally satisfies simplex constraints—ensuring that asset weights are always non-negative and sum to exactly 100%.

Experimental Setup

The model was evaluated using historical market data, incorporating price, volume, and various financial indicators calculated over rolling windows (1 to 12 months). The simulation environment factored in a 0.25% transaction fee per trade to ensure realistic performance metrics. The primary objective was to maximize returns while maintaining zero violations of the pre-defined risk constraints.

Results and Discussion

The "Proposed ALM" model was benchmarked against several established RL algorithms, including PPO, TRPO, and Bayesian Policy Gradient (BPG). The results demonstrate that the ALM approach successfully bridges the gap between safety and performance.

Comparative performance of the Proposed ALM against baseline RL models.

As shown in the performance summary, while a standard DDPG agent achieved the highest raw annual return (40.3%), it did so by violating risk constraints 154 times. In contrast, the Proposed ALM achieved a competitive 35.7% return with zero constraint violations. This highlights the model's ability to internalize safety requirements as rigid boundaries rather than mere suggestions.

Ablation Study

An ablation study confirmed the necessity of the "Augmented" components. Removing either the Lagrangian multiplier or the quadratic penalty resulted in a significant performance degradation, with the most basic version of the model achieving only 89% of the full system's effectiveness. This underscores that both components are essential for maintaining the balance between reward seeking and constraint satisfaction.

Conclusion

This research demonstrates that risk-constrained RL can achieve professional-grade safety standards in portfolio optimization. By integrating a dedicated Cost Network and an Augmented Lagrangian objective, the framework provides a robust solution for investors who require strict adherence to risk limits. Future work may explore the model's scalability to larger asset universes and its resilience during extreme "black swan" market events.

Summary updated

BibTeX
@inproceedings{enkhsaikhan2025augmented,
  author = {Bayaraa Enkhsaikhan and Ohyun Jo},
  title = {{Augmented Lagrangian Risk-constrained Reinforcement Learning for Portfolio Optimization (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35252},
}
Figure