Abstract
We analyze a fixed-point algorithm for reinforcement learning (RL) of optimal portfolio mean-variance preferences in the setting of multivariate generalized autoregressive conditional-heteroskedasticity (MGARCH) with a small penalty on trading. A numerical solution is obtained using a neural network (NN) architecture within a recursive RL loop. A fixed-point theorem proves that NN approximation error has a big-oh bound that we can reduce by increasing the number of NN parameters. The functional form of the trading penalty has a parameter $\epsilon > 0$ that controls the magnitude of transaction costs. When $\epsilon$ is small, we can implement an NN algorithm based on the expansion of the solution in powers of $\epsilon$. This expansion has a base term equal to a myopic solution with an explicit form, and a first-order correction term that we compute in the RL loop. Our expansion-based algorithm is stable, allows for fast computation, and outputs a solution that shows positive testing performance.
Introduction
Portfolio management in real-world markets requires navigating time-varying volatility (heteroskedasticity) and the friction of transaction costs. While classic mean-variance optimization provides a baseline, it often fails to account for the dynamic nature of market risk and the costs associated with rebalancing. This paper addresses the challenge of optimizing a multi-asset portfolio under a Multivariate GARCH (MGARCH) process, where the optimal strategy lacks a simple closed-form solution due to constantly changing coefficients. Traditional reinforcement learning (RL) methods, such as actor-critic architectures, often suffer from instability and high computational costs in these high-dimensional financial environments.
Methodology
The proposed framework combines control theory with deep learning to create a stable and efficient RL algorithm. The approach is built on four key pillars:
- Linear-Quadratic Framework: The problem is formulated as a linear-quadratic program that penalizes both trading activity and portfolio risk.
- Asymptotic Expansion: By assuming the transaction cost parameter ($\epsilon$) is small, the researchers expand the optimal solution into a power series. This consists of a "myopic" base term (current-moment optimization) and a first-order correction term that accounts for future expectations.
- Fixed-Point Iteration: To ensure stability, the algorithm utilizes a fixed-point iteration rather than standard gradient-based RL. This method is mathematically proven to keep neural network approximation errors bounded.
- Neural Network Architecture: A deep neural network (DNN) with 4 hidden layers of 400 neurons each is employed. Crucially, the network only learns the correction term relative to the myopic baseline, significantly improving training speed and reliability.
Key Insights and Results
A central finding of the study is the relationship between market stress and the "condition number" of the covariance matrix. As market volatility increases and the degrees of freedom in the covariance matrix decrease, the condition number spikes. The algorithm responds by automatically increasing the penalty for trading, leading to more conservative behavior during periods of high stress.
The algorithm's performance was validated using both synthetic and historical datasets:
- Synthetic Data: The RL strategy achieved an annual increase of 1.8% in total value over the myopic baseline.
- Historical Data: Using 11 sector ETFs from 2010 to 2019, the RL method consistently outperformed the myopic approach by an average of 11 basis points (bps).
- Efficiency: The structured approach allows for extremely fast inference, taking approximately 1.5 milliseconds on standard hardware, making it viable for real-time applications.
Conclusion
By bridging the gap between simple myopic trading and complex long-term strategies through asymptotic expansion, this research provides a robust framework for portfolio optimization. While the current model relies on the assumption of small transaction costs and an MGARCH structure, it demonstrates a significant improvement in stability and performance over "black-box" RL methods. Future work may explore higher-order expansions for high-friction markets and adaptation to non-GARCH market regimes.
Summary updated