TL;DR
This paper introduces a novel two-level reinforcement learning framework to tackle the challenging problem of mean-variance portfolio selection when the underlying asset return distributions are ambiguous. By separating the handling of return randomness from the uncertainty in distribution proportions, their method learns an optimal investment strategy in a dynamic, time-varying market.
What Problem Does This Paper Address?
Traditional portfolio selection models, like Markowitz's mean-variance framework, often assume investors have perfect knowledge of asset return statistics (means and covariances). However, real-world financial markets are dynamic and uncertain, making this assumption unrealistic. Specifically, this paper focuses on the ambiguity arising from a mixture model of asset returns, where the investor knows the potential underlying distributions (e.g., bull, bear, neutral market modes) but is uncertain about their exact proportions. This uncertainty is crucial because it's reducible through learning from market observations. The challenge is that incorporating this "learning feature" into traditional dynamic programming or optimization methods becomes computationally intractable due to the non-linear nature of updating these proportions.
Methods
The authors propose a two-level reinforcement learning (RL) framework to address this intractability.
- Lower Level (Time-Decomposed DP): For each specific scenario (a fixed sequence of component distributions over time), they use dynamic programming (DP) to solve a family of subproblems. This effectively handles the randomness of returns given a known set of component distributions.
- Upper Level (Scenario-Decomposed Progressive Hedging Algorithm - PHA): This level iteratively aggregates the solutions from the lower level. It uses the Progressive Hedging Algorithm (PHA), a method designed for multi-scenario optimization problems, to combine scenario-specific policies into an implementable one. This aggregation process is guided by the current knowledge of the ambiguous proportions.
The entire process operates on a rolling horizon, meaning the problem is solved for a truncated future period, and then the framework "rolls" forward in time, updating information and re-solving. Crucially, the learning of proportions is handled by an Expectation-Maximization (EM) algorithm (Algorithm 1), which is applied at each time step to update the investor's belief about the mixture proportions based on observed returns.
Datasets
Not discussed in the paper. The experimental study uses simulated data generated from a Gaussian Mixture Model (GMM).
Key Insights and Ideas
- Decoupling Ambiguity: The core innovation is separating the problem into two levels: one for return randomness (DP) and one for proportion uncertainty (PHA with EM learning). This makes the problem tractable.
- Active Learning: The framework inherently incorporates an "active learning" feature by continuously updating the investor's belief about the mixture proportions as new market data becomes available. This is essential for adapting to changing market conditions.
- Rolling Horizon for Adaptability: The use of a rolling horizon allows the portfolio strategy to adapt over time, re-evaluating decisions based on the most current information and learned proportions.
- Two-Level Structure: The synergy between DP (for scenario-specific solutions) and PHA (for aggregating across scenarios and handling ambiguity) is key to the algorithm's effectiveness.
Key Results
The paper presents an experimental study (Example 1) with a simulated market of three risky assets and one risk-free asset over 40 periods.
- Learning Effectiveness: Figure 2 demonstrates that the EM algorithm effectively learns the true proportions of the mixture model over time, both during a pre-investing period and during the rolling horizon.
- Policy Improvement: Figure 3 shows that the proposed Two-level Learning (TL) policy yields coefficients (K and V) that are significantly closer to the Full-Knowledge (FK) policy (which assumes perfect knowledge) compared to a Non-Learning (NL) policy. The NL policy, which uses initial beliefs without updating, accumulates errors.
- Performance Fluctuations: The TL policy's performance is shown to be sensitive to the accuracy of the learned proportions. Its closest approximation to the FK policy occurs when the proportion estimation is at its best.
Limitations and Future Work
- Sensitivity to Learning: The paper acknowledges that the TL policy's performance can be affected by the fluctuations in the learned proportions.
- Future Work: The authors suggest improving the framework by incorporating "more active learning aspects," specifically by considering all possible future learning outcomes in advance when solving the approximated primal problem at time 0. This implies a more sophisticated lookahead mechanism.
Summary updated