This paper, presented at ICAIF '26, introduces a novel way to bridge the gap between high-performing deep reinforcement learning (RL) and the strict, rule-based world of institutional finance. The authors propose PLPG-MC, a neurosymbolic approach that allows RL agents to "internalize" complex investment mandates directly into their policy, drastically reducing rule violations in continuous action spaces.
TL;DR
The authors develop a continuous-action neurosymbolic RL framework (PLPG-MC) that uses reparameterized Monte Carlo sampling to estimate the probability of rule compliance during training. By adding a differentiable symbolic safety loss to the objective, they reduce mandate violations from over 96% in standard RL to just 4.8%, without requiring post-hoc action correction at inference time.
What Problem Does This Paper Address?
Institutional portfolios aren't just about maximizing returns; they must follow an investment policy statement (mandate). These mandates consist of symbolic rules—like "don't hold more than 80% equities during a bull market"—that are often conditioned on the current market regime.
Standard Deep RL agents are "reward-maximizers" with no inherent understanding of these symbolic boundaries. While researchers often use Constrained MDPs (CMDPs) or reward penalties to handle constraints, these methods treat violations as a "cost" that can be traded off for higher returns. In the real world, a mandate breach is a hard failure, regardless of profit.
Existing neurosymbolic methods like Probabilistic Logic Policy Gradient (PLPG) can enforce hard rules, but they typically assume discrete action spaces. Portfolio allocation, however, is continuous (choosing weights for a set of assets), making the math behind these symbolic shields intractable.
Methods
The authors propose a framework that combines neural learning (PPO) with symbolic logic. The key components are:
- HMM Market-Regime Detection: A Hidden Markov Model identifies three market states: Bull, Bear, and Crisis. These regimes determine which specific thresholds in the mandate are active (see Figure 5 for the mandate details).
- PLPG-MC (Monte Carlo): To handle continuous actions, the authors replace the discrete normalizer used in traditional PLPG with a reparameterized Monte Carlo estimate. During training, the agent draws $K$ samples from its current policy and evaluates them against the symbolic rules.
- Differentiable Symbolic Regularization: They define "signed slack"—a measure of how far an action is from a constraint boundary. This slack is passed through a logistic sigmoid to create a differentiable "safety loss" ($L_{safe}$).
- Sigma-Detach: To prevent the agent from simply "cheating" by shrinking its exploration variance to zero to satisfy the safety loss, the authors stop the gradient flow through the policy's standard deviation ($\sigma$) for the safety term.
The overall architecture is summarized in Figure 1:
Datasets
The study uses daily market data from June 2005 to May 2026 (including synthetic/projected data for the later years).
- Assets: 10 ETFs covering equities (SPY, QQQ, etc.), fixed income (TLT, IEF, LQD), and commodities (GLD, SLV), plus the VIX.
- Training/Test Split: The model is trained on data through 2020 and evaluated on a held-out test period (2021–2026) that includes significant stress events like the 2022 drawdown and the 2025 "tariff shock."
The HMM's ability to capture these stress periods is visualized in Figure 2:
Key Insights and Ideas
- Internalization vs. Shielding: Unlike "shielding" methods that intercept and correct actions at runtime, this approach teaches the policy to be safe inherently. This reduces "projection distortion," where an agent's intent is constantly fighting against an external safety layer.
- The "Safe-Harbor Effect": The authors discovered that if safe regions for different regimes overlap significantly, the agent might find a single "safe harbor" allocation that satisfies all regimes, causing it to stop responding to market changes. They argue that mandate geometry is a critical, often overlooked factor in regime-aware RL.
- Direct Compliance Audit: The policy is audited on its deterministic mean actions with no post-hoc corrections. This is a much stricter test than typical constrained RL evaluations.
Key Results
The main results (Table 2) show a massive improvement in compliance:
- Violation Rate (PV): Unconstrained PPO violates the mandate nearly every day (~97%). A standard reward penalty reduces this to 33.4%. PLPG-MC achieves a 4.8% violation rate.
- Risk Mitigation: While the constrained agent earns lower total returns (~41% vs ~100% for unconstrained), it does so with significantly lower volatility and maximum drawdown.
- Dose-Response: As shown in Figure 3, increasing the safety weight ($\lambda$) quickly drives down violations to a stable floor.
Limitations and Future Work
- Residual Violations: The remaining 4.8% of violations are largely due to detector lag—the HMM takes a few days to realize a regime has changed, during which time the agent is following the "old" rules.
- Price-Taker Assumption: The model assumes the agent's trades don't move market prices, which may not hold for very large institutional portfolios.
- Scope: The current work does not explicitly include transaction costs or turnover penalties, though the authors note these could be added as additional symbolic rules.
- Future Directions: The authors suggest extending this to distributional risk constraints (like CVaR) and allowing mandate thresholds to evolve dynamically with market state.
Summary updated