A Novel Experts Advice Aggregation Framework Using Deep Reinforcement Learning for Portfolio Management

MohammadAmin Fazli, Mahdi Lashkari, Hamed Taherkhani, Jafar HabibiarXiv 2022 · DOI · arXiv

Introduction

Portfolio management is the complex process of continuously reallocating capital across various assets to maximize returns while mitigating risk. Traditionally, automated trading systems have relied on two primary pillars: technical analysis (historical price patterns) and fundamental analysis (economic indicators and earnings). However, financial markets are also deeply influenced by human sentiment and expert intuition—data that is notoriously difficult to quantify and integrate into algorithmic models.

This paper introduces a Deep Reinforcement Learning (DRL) framework designed to bridge this gap. By treating expert predictions not as absolute commands but as high-level data features, the model learns to intelligently fuse human intuition with raw market data to navigate dynamic, non-deterministic financial environments.

Methodology

Multi-Stream Architecture

The authors propose a dual-stream neural network architecture to process disparate data types simultaneously:

  • Price Stream: A Convolutional Neural Network (CNN) processes 60 days of historical Open, High, Low, Close, and Volume (OHLCV) data to capture short-term market trends.
  • Expert Stream: A parallel CNN processes 60 days of "expert signals." These signals include structured data such as expected returns, projected risk, and signal status. When multiple experts provide conflicting signals for the same asset, the model uses an average of the expected profit/loss to de-noise the input.
  • Feature Fusion: The outputs from both CNNs are concatenated with the agent's previous portfolio weight vector. This inclusion is critical; by "knowing" its current holdings, the agent learns to avoid excessive trading (churn) that would otherwise be eroded by transaction fees.

The PPO Agent

The framework utilizes Proximal Policy Optimization (PPO), an Actor-Critic algorithm, within a custom environment built on the Ray and Rllib frameworks. PPO is particularly effective in financial contexts because it constrains the magnitude of policy updates. This prevents the model from overreacting to temporary market shocks, leading to more stable training and reliable performance in volatile conditions.

Experimental Setup

The study focused on the Tehran Stock Exchange (Iran) using data from 2017 to 2020. The dataset included:

  • Daily OHLCV data for 54 distinct stocks.
  • 6,335 signals from 85 experts sourced from the 100tahlil financial platform.

Robust Training Constraints

To instill a sense of risk management, the authors implemented a strict "Stop-Loss" training rule. If a portfolio's value dropped by 10% from the episode start or 20% from its peak, the training episode was immediately terminated. This penalty forced the RL agent to prioritize capital preservation alongside growth.

Results and Discussion

The framework demonstrated that a machine-learning model can effectively "stand on the shoulders of giants" by filtering and weighting expert advice. Key performance metrics include:

  • Profit Capture: The model successfully captured 90% of the profit achieved by the single best-performing human expert in the dataset.
  • Benchmark Comparison: During the six-month test period, the top human expert averaged a 72% profit, while the bottom-tier experts suffered losses of up to 8%. The DRL model achieved an average gain of 65%, with peak performance reaching 85%.
  • Operational Efficiency: By accounting for a 0.1% transaction fee and incorporating previous portfolio states, the model learned a sophisticated rebalancing strategy that minimized unnecessary costs while remaining responsive to new expert signals.

Conclusion and Future Directions

This research proves that expert advice can be effectively aggregated using DRL to outperform traditional weighted-average methods. By treating experts as features rather than masters, the agent learns which advisors are reliable under specific market conditions.

Future iterations of this work could expand the model's utility by:

  1. Integrating NLP: Using Natural Language Processing to ingest unstructured expert advice from news articles and social media.
  2. Modeling Slippage: Incorporating more complex market impact models to account for the difference between expected and executed trade prices.
  3. Cross-Market Validation: Testing the framework on higher-liquidity global exchanges like the NYSE to determine if the strategy holds across different regulatory and economic environments.

Summary updated

BibTeX
@misc{fazli2022novel,
  author = {MohammadAmin Fazli and Mahdi Lashkari and Hamed Taherkhani and Jafar Habibi},
  title = {{A Novel Experts Advice Aggregation Framework Using Deep Reinforcement Learning for Portfolio Management}},
  year = {2022},
  eprint = {2212.14477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.14477},
}
Figure