Contrastive Learning and Reward Smoothing for Deep Portfolio Management

Yun-Hsuan Lien, Yuan-kui Li, Yu-Shuen Wang

Introduction

Deep Reinforcement Learning (DRL) has emerged as a powerful tool for portfolio management (PM), the process of reallocating assets to maximize returns while managing risk. However, financial markets present unique challenges: they are notoriously noisy, non-stationary, and unpredictable. Standard DRL agents often overfit to transient price spikes in training data, leading to poor generalization in live trading environments.

This paper addresses these hurdles by introducing a novel framework that integrates Neural Relational Inference (NRI) for contrastive learning and reward smoothing (sRT). By grouping assets with similar future trends and discouraging aggressive, short-sighted trading, the proposed model achieves a more robust and stable strategy across diverse market conditions.

Methodology: Robust Representation and Temporal Regularization

The authors propose a multi-task learning objective that balances three goals: immediate returns, long-term stability, and robust representation learning.

Positive Pair Matching (PPM) via NRI

A significant challenge in applying contrastive learning to time-series data is the lack of intuitive data augmentations. Unlike computer vision, where flipping or cropping an image preserves its semantic meaning, augmenting a price trajectory can destroy critical temporal patterns.

To overcome this, the authors utilize Positive Pair Matching (PPM). Instead of manual augmentation, they use Neural Relational Inference (NRI)—a Graph Neural Network (GNN) that discovers latent relationships between assets. If two assets are predicted to have highly correlated future price movements, they are identified as a "positive pair." A contrastive loss then ensures that the internal representations of these paired assets are close in vector space, forcing the model to learn underlying market dynamics rather than memorizing individual noise.

Conceptual overview of Positive Pair Matching (PPM) using future price trends to identify latent relationships between assets.

Reward Smoothing (sRT)

Standard DRL agents are often biased toward aggressive, high-frequency trading to capture immediate gains. To counter this, the authors introduce reward smoothing (sRT). This component acts as a temporal regularizer by encouraging actions that remain profitable over a future window of time steps.

The importance of this smoothed reward is dynamically weighted based on the agent's current performance. When the agent is underperforming, the model prioritizes long-term stability to help it find a more sustainable strategy.

Experimental Evaluation

The framework was evaluated on two distinct datasets:

  1. U.S. Stock Market: Daily data for nine high-volume stocks (e.g., GOOG, NVDA, AAPL) from 2001 to 2013.
  2. Cryptocurrency Market: Half-hour frequency data for ten major cryptocurrencies (e.g., ETH, LTC, XRP) from 2015 to 2018.

Performance was tested across three market conditions: Increase, No change, and Decrease.

Key Results and Insights

The proposed method, Ours (PPM+sRT), consistently outperformed traditional baselines (Buy & Hold, UCRP) and state-of-the-art DRL models (SARL, DPM).

Portfolio Value (PV) over time across different market scenarios, demonstrating the superior performance and stability of the proposed PPM+sRT method.

Performance Highlights

  • Market Resilience: The model showed exceptional resilience during market "Decrease" scenarios. In the cryptocurrency market, it achieved a Portfolio Value (PV) significantly higher than the standard DPM model.
  • Risk-Adjusted Returns: Ablation studies confirmed that the combination of PPM and sRT significantly boosts the Sharpe Ratio, indicating better risk-adjusted performance in volatile environments.
  • Dynamic Relationships: By using NRI, the model automatically adapts to changing market correlations, moving beyond static heuristic rules like "tech stocks move together."

Conclusion

By treating forecasting as a pretext task for contrastive learning and implementing reward smoothing, this research provides a path toward more reliable DRL agents in finance. While the model achieves high returns, future work could focus on explicitly minimizing maximum drawdown (MDD) to better serve risk-averse institutional investors. Additionally, bridging the gap between historical back-testing and real-world execution remains a critical next step.

Summary updated

Abstract

In this study, we used reinforcement learning (RL) models to invest assets in order to earn returns. The models were trained to interact with a simulated environment based on historical market data and learn trading strategies. However, using deep neural networks based on the returns of each period can be challenging due to the unpredictability of financial markets. As a result, the policies learned from training data may not be effective when tested in real-world situations. To address this issue, we incorporated contrastive learning and reward smoothing into our training process. Contrastive learning allows the RL models to recognize patterns in asset states that may indicate future price movements. Reward smoothing, on the other hand, serves as a regularization technique to prevent the models from seeking immediate but uncertain profits. We tested our method against various traditional financial techniques and other deep RL methods, and found it to be effective in both the U.S. stock market and the cryptocurrency market. Our source code is available at https://github.com/sophialien/FinTech-DPM.
BibTeX
@inproceedings{lien2023contrastive,
  author = {Yun-Hsuan Lien and Yuan-kui Li and Yu-Shuen Wang},
  title = {{Contrastive Learning and Reward Smoothing for Deep Portfolio Management}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/441},
}
Figure