A Universal End-to-End Approach to Portfolio Optimization via Deep\n Learning

Chao Zhang, Zihao Zhang, Mihai Cucuringu, Stefan ZohrenarXiv 2021 · DOI · arXiv

Abstract: We propose a universal end-to-end framework for portfolio optimization where asset distributions are directly obtained. The designed framework circumvents the traditional forecasting step and avoids the estimation of the covariance matrix, lifting the bottleneck for generalizing to a large amount of instruments. Our framework has the flexibility of optimizing various objective functions including Sharpe ratio, mean-variance trade-off etc. Further, we allow for short selling and study several constraints attached to objective functions. In particular, we consider cardinality, maximum position for individual instrument and leverage. These constraints are formulated into objective functions by utilizing several neural layers and gradient ascent can be adopted for optimization. To ensure the robustness of our framework, we test our methods on two datasets. Firstly, we look at a synthetic dataset where we demonstrate that weights obtained from our end-to-end approach are better than classical predictive methods. Secondly, we apply our framework on a real-life dataset with historical observations of hundreds of instruments with a testing period of more than 20 years.

Introduction

Modern Portfolio Theory (MPT) has long been the cornerstone of quantitative finance. However, its practical application is often hindered by two significant bottlenecks: the high noise-to-signal ratio in financial data and the computational complexity of estimating covariance matrices for large asset universes. Traditional methods typically follow a two-step "predict-then-optimize" pipeline. Unfortunately, small errors in return forecasting are amplified during the optimization phase, leading to unstable and often suboptimal portfolio weights—a phenomenon known as the "Markowitz optimization enigma."

The End-to-End Framework

To address these challenges, we introduce a universal end-to-end (E2E) deep learning framework. Unlike traditional methods, our approach directly maps raw market data to optimal portfolio weights, bypassing the intermediate forecasting and covariance estimation steps. The architecture consists of two primary components: the Score Block and the Portfolio Block.

Architecture of the proposed end-to-end portfolio optimization framework.

The Score Block ($h_1$) acts as a feature extractor, utilizing neural network architectures such as Multi-Layer Perceptrons (MLPs), Long Short-Term Memory (LSTM) networks, or Convolutional Neural Networks (CNNs) to generate "fitness scores" for each asset. These scores represent the desirability of an asset relative to the chosen objective function.

The Portfolio Block ($h_2$) is the core innovation. It transforms the fitness scores into portfolio weights ($w_{i,t}$) while ensuring that all financial constraints are strictly satisfied. By using differentiable layers, the entire system can be trained via gradient ascent to maximize specific risk-adjusted performance metrics, such as the Sharpe Ratio or mean-variance utility.

Differentiable Constraints and Scalability

One of the most significant advantages of this framework is its ability to handle complex, non-convex constraints that are traditionally difficult to solve. For instance, institutional investors often face cardinality constraints (limiting the number of stocks) or position limits. We incorporate these by utilizing specialized neural layers, such as NeuralSort—a differentiable sorting algorithm—which allows the model to rank and select assets while maintaining a continuous gradient flow for training. This approach makes the framework highly scalable, capable of managing hundreds of instruments simultaneously without the numerical instability associated with large covariance matrices.

Experimental Results

The framework was rigorously tested on both synthetic and real-world datasets. The real-world evaluation utilized the WRDS dataset, comprising 735 stocks from the Russell 3000 Index over a 37-year period (1984–2021), with a 20-year testing window.

Performance comparison of the E2E framework against various baselines and constraints on the WRDS dataset.

As shown in the results table, the E2E-LSTM model significantly outperformed traditional benchmarks. It achieved an annualized Sharpe Ratio of 2.604, far exceeding the S&P 500 (0.402) and the classical two-step predictive method (CS-LSTM, 0.858). Furthermore, the model demonstrated remarkable robustness, maintaining stable performance through major market disruptions like the 2008 financial crisis and the 2020 pandemic.

The framework also proved its versatility in handling multiple simultaneous constraints. The "LSTM-MSRP-LEV-CAR-MAX" model successfully managed leverage, cardinality, and maximum position limits while still delivering competitive risk-adjusted returns, proving the framework's readiness for institutional requirements.

Conclusion

By shifting from a predictive paradigm to a direct optimization framework, we provide a robust solution to the classic pitfalls of portfolio management. The end-to-end approach not only improves performance but also offers a flexible, scalable way to integrate complex real-world trading requirements into a unified deep learning model. Future work will focus on integrating transaction costs directly into the objective function and exploring reinforcement learning techniques to further enhance dynamic portfolio rebalancing.

Summary updated

Abstract

We propose a universal end-to-end framework for portfolio optimization where\nasset distributions are directly obtained. The designed framework circumvents\nthe traditional forecasting step and avoids the estimation of the covariance\nmatrix, lifting the bottleneck for generalizing to a large amount of\ninstruments. Our framework has the flexibility of optimizing various objective\nfunctions including Sharpe ratio, mean-variance trade-off etc. Further, we\nallow for short selling and study several constraints attached to objective\nfunctions. In particular, we consider cardinality, maximum position for\nindividual instrument and leverage. These constraints are formulated into\nobjective functions by utilizing several neural layers and gradient ascent can\nbe adopted for optimization. To ensure the robustness of our framework, we test\nour methods on two datasets. Firstly, we look at a synthetic dataset where we\ndemonstrate that weights obtained from our end-to-end approach are better than\nclassical predictive methods. Secondly, we apply our framework on a real-life\ndataset with historical observations of hundreds of instruments with a testing\nperiod of more than 20 years.\n
BibTeX
@misc{zhang2021universal,
  author = {Chao Zhang and Zihao Zhang and Mihai Cucuringu and Stefan Zohren},
  title = {{A Universal End-to-End Approach to Portfolio Optimization via Deep\textbackslash\{\}n Learning}},
  year = {2021},
  eprint = {2111.09170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.09170},
}
Figure