Deep Portfolio Optimization via Distributional Prediction of Residual Factors

Kentaro Imajo, Kentaro Minami, Katsuya Ito, Kei NakagawaAAAI 2021 · DOI

Abstract

Predicting financial time series is notoriously difficult due to non-stationarity and the dominance of broad market factors. This paper introduces Deep Portfolio Optimization (DPO), a framework that integrates financial theory with deep learning. By extracting "spectral residuals" to isolate idiosyncratic stock behavior and employing "fractal networks" for distributional prediction, DPO achieves superior risk-adjusted returns. Experimental results over a 20-year period on the S&P 500 demonstrate that DPO significantly outperforms traditional and state-of-the-art machine learning baselines in both Sharpe ratio and volatility reduction.

Introduction

Standard machine learning models often struggle in finance because they lack the specific inductive biases required to handle market noise and shifting regimes. Two major hurdles identified are market factor dominance—where stocks move in unison based on broad trends—and the necessity of predicting the full probability distribution of returns for effective risk management, rather than just the mean.

Methodology

The DPO framework addresses these challenges through a three-part system designed to isolate stock-specific opportunities and model uncertainty.

The DPO framework architecture, illustrating the flow from raw returns to spectral residuals, distributional prediction, and final portfolio weights.

1. Spectral Residual Extraction

To isolate stock-specific information, the authors project raw returns onto a space spanned by the smallest eigenvalues of the return covariance matrix using PCA. These "spectral residuals" effectively hedge out common market factors. This method is computationally efficient, taking only 10 minutes to process 5,000 days of data, compared to 13 hours for traditional Factor Analysis (FA).

2. Distributional Prediction via Quantile Regression

Instead of point estimates, the model uses a neural network to predict multiple conditional quantiles of the residuals. This allows the system to estimate both the expected mean ((\mu)) and the variance ((\sigma^2)), which are then utilized in the classic Markowitz portfolio optimization formula.

3. Fractal Network Architecture

The predictive model incorporates a novel architecture with two key financial biases:

  • Volatility Invariance: The network is designed to be amplitude-invariant, treating percentage moves consistently regardless of the current volatility regime.
  • Time-scale Invariance: Inspired by the fractal nature of markets, the network resamples input data at different resolutions to capture self-similar patterns across various time horizons.

Experimental Results

The researchers evaluated DPO on the U.S. (S&P 500) and Japanese markets from 2000 to 2020. The model was trained on data prior to 2008 and tested out-of-sample using daily opening prices to ensure realistic executability.

Performance metrics for the U.S. market (S&P 500), comparing DPO against market benchmarks and various machine learning baselines.

As shown in the results table, DPO achieved a remarkable Annualized Sharpe Ratio (ASR) of +1.393, significantly outperforming the Market (+0.607) and the best ML baseline (SFM at +0.709). DPO also demonstrated superior risk management, maintaining the lowest Annualized Volatility (AVOL) and a high Downside Deviation Ratio (DDR). Ablation studies (DPO-NQ, -NF, -NV) confirm that both the distributional prediction and the fractal architectural choices are essential for these performance gains.

Conclusion

Deep Portfolio Optimization successfully bridges the gap between financial domain knowledge and deep learning. By focusing on spectral residuals and distributional forecasting, the model provides a robust strategy that remains stable even during major market crises. Future work will focus on modeling transaction costs and applying these techniques to broader asset classes beyond equities.

Summary updated

BibTeX
@inproceedings{imajo2021deep,
  author = {Kentaro Imajo and Kentaro Minami and Katsuya Ito and Kei Nakagawa},
  title = {{Deep Portfolio Optimization via Distributional Prediction of Residual Factors}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i1.16095},
}
Figure