Introduction
Deep Reinforcement Learning (DRL) has emerged as a powerful tool for portfolio management (PM), the process of reallocating assets to maximize returns while managing risk. However, financial markets present unique challenges: they are notoriously noisy, non-stationary, and unpredictable. Standard DRL agents often overfit to transient price spikes in training data, leading to poor generalization in live trading environments.
This paper addresses these hurdles by introducing a novel framework that integrates Neural Relational Inference (NRI) for contrastive learning and reward smoothing (sRT). By grouping assets with similar future trends and discouraging aggressive, short-sighted trading, the proposed model achieves a more robust and stable strategy across diverse market conditions.
Methodology: Robust Representation and Temporal Regularization
The authors propose a multi-task learning objective that balances three goals: immediate returns, long-term stability, and robust representation learning.
Positive Pair Matching (PPM) via NRI
A significant challenge in applying contrastive learning to time-series data is the lack of intuitive data augmentations. Unlike computer vision, where flipping or cropping an image preserves its semantic meaning, augmenting a price trajectory can destroy critical temporal patterns.
To overcome this, the authors utilize Positive Pair Matching (PPM). Instead of manual augmentation, they use Neural Relational Inference (NRI)—a Graph Neural Network (GNN) that discovers latent relationships between assets. If two assets are predicted to have highly correlated future price movements, they are identified as a "positive pair." A contrastive loss then ensures that the internal representations of these paired assets are close in vector space, forcing the model to learn underlying market dynamics rather than memorizing individual noise.
Reward Smoothing (sRT)
Standard DRL agents are often biased toward aggressive, high-frequency trading to capture immediate gains. To counter this, the authors introduce reward smoothing (sRT). This component acts as a temporal regularizer by encouraging actions that remain profitable over a future window of time steps.
The importance of this smoothed reward is dynamically weighted based on the agent's current performance. When the agent is underperforming, the model prioritizes long-term stability to help it find a more sustainable strategy.
Experimental Evaluation
The framework was evaluated on two distinct datasets:
- U.S. Stock Market: Daily data for nine high-volume stocks (e.g., GOOG, NVDA, AAPL) from 2001 to 2013.
- Cryptocurrency Market: Half-hour frequency data for ten major cryptocurrencies (e.g., ETH, LTC, XRP) from 2015 to 2018.
Performance was tested across three market conditions: Increase, No change, and Decrease.
Key Results and Insights
The proposed method, Ours (PPM+sRT), consistently outperformed traditional baselines (Buy & Hold, UCRP) and state-of-the-art DRL models (SARL, DPM).
Performance Highlights
- Market Resilience: The model showed exceptional resilience during market "Decrease" scenarios. In the cryptocurrency market, it achieved a Portfolio Value (PV) significantly higher than the standard DPM model.
- Risk-Adjusted Returns: Ablation studies confirmed that the combination of PPM and sRT significantly boosts the Sharpe Ratio, indicating better risk-adjusted performance in volatile environments.
- Dynamic Relationships: By using NRI, the model automatically adapts to changing market correlations, moving beyond static heuristic rules like "tech stocks move together."
Conclusion
By treating forecasting as a pretext task for contrastive learning and implementing reward smoothing, this research provides a path toward more reliable DRL agents in finance. While the model achieves high returns, future work could focus on explicitly minimizing maximum drawdown (MDD) to better serve risk-averse institutional investors. Additionally, bridging the gap between historical back-testing and real-world execution remains a critical next step.
Summary updated