Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization

Haochen Yuan, Minting Pan, Yunbo Wang, Siyu Gao, Philip S. Yu, Xiaokang YangarXiv 2025 · DOI · arXiv

Introduction

Reinforcement learning (RL) has shown promise in financial markets, yet it frequently fails in real-world deployment. The primary hurdle is the generalization-optimality dilemma: agents trained on historical data often overfit to specific past market conditions, failing to generalize when the underlying distribution shifts. Standard offline RL also suffers from value overestimation, where agents become overly optimistic about actions they haven't explored, leading to risky financial decisions.

To address these challenges, this paper introduces MetaTrader, a "partial-offline" RL framework. MetaTrader utilizes bilevel optimization and synthetic data transformations to train agents capable of navigating unseen, out-of-distribution (OOD) market scenarios.

The MetaTrader framework addresses the gap between training and test distributions by simulating OOD scenarios through data transformations and bilevel training.

Methodology

MetaTrader reframes portfolio optimization through several innovative technical components:

Partial-Offline RL Formulation

The authors decouple the environment's state into two distinct branches:

  1. Market States ($h_t$): Global indicators like stock prices and trends. These are treated as action-free and fixed to the offline dataset, as an individual trader's actions do not impact the global market.
  2. Balance States ($z_t$): The agent's specific holdings (cash and stock). These are action-dependent. This decoupling allows the agent to safely explore various portfolio allocations over fixed historical market data without needing to model the complex dynamics of the global market itself.

Market Data Transformations

To prevent memorization and simulate OOD conditions, MetaTrader generates synthetic data using three primary transformations:

  • Unexpected Events ($F_1$): Inverting the growth rates of top-performing stocks to simulate sudden market crashes.
  • Chronological Reversal ($F_2$): Reversing trends to simulate long-term shifts in market regime.
  • Downsampling ($F_3$): Capturing multi-scale dynamics by reducing data frequency.
MetaTrader applies three transformations—unexpected events, chronological reversal, and downsampling—to historical data to create diverse OOD training scenarios.

Bilevel Policy Learning and Conservative TD

MetaTrader employs a meta-learning loop. The inner loop trains the policy to maximize profits on standard historical data. The outer loop evaluates and updates parameters against the transformed OOD data, acting as a powerful regularizer that prioritizes safe, generalizable strategies.

Furthermore, the framework introduces Transformation-based Temporal Difference (TD) Learning. To combat value overestimation, the agent computes next-step Q-values for both original and transformed data, using the minimum (worst-case) value as its target. This conservative approach proves more efficient and effective than traditional ensemble-based RL.

Experimental Evaluation

The framework was evaluated on two major public stock datasets: CSI-300 (Chinese market) and NASDAQ-100 (US tech stocks), covering the period from 2011 to 2022.

Key Results

MetaTrader significantly outperformed traditional forecasting models (e.g., FactorVAE) and state-of-the-art RL baselines (e.g., StockFormer, FinRL):

  • CSI-300 Performance: MetaTrader achieved a Cumulative Return (CR) of 1.44, a 50% improvement over FactorVAE (0.96 CR) and a 16.1% improvement over StockFormer (1.24 CR).
  • Online Adaptation: When fine-tuned on streaming test data, MetaTrader's performance on the NASDAQ-100 improved cumulative returns by over 25% compared to StockFormer (1.58 vs 1.26) and increased the Sharpe ratio by approximately 43%.
  • Ablation Study: The custom worst-case TD learning module alone contributed a 9.5% boost in cumulative returns compared to standard SAC TD learning.

Discussion and Future Work

MetaTrader demonstrates that data diversity, achieved through targeted transformations, can be more effective than model diversity in conservative RL. By separating uncontrollable market states from controllable balance states, the framework bypasses the difficulty of state estimation in offline RL.

However, challenges remain. RL-based methods like MetaTrader exhibit higher training variance compared to supervised models. Future research will focus on stabilizing training through better initialization and extending the framework to high-frequency trading scenarios. Additionally, the "partial-offline" MDP formulation holds potential for other non-stationary domains, such as autonomous driving.

Summary updated

Abstract

Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data. However, traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset. These offline policies are less generalizable as they fail to account for the non-stationary nature of the market. Our approach, MetaTrader, frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions. First, MetaTrader employs a bilevel learning framework that explicitly trains the RL agent to improve both in-domain profits on the original dataset and out-of-domain performance across diverse transformations of the raw financial data. Second, our approach incorporates a new temporal difference (TD) method that approximates worst-case TD estimates from a batch of transformed TD targets, addressing the value overestimation issue that is particularly challenging in scenarios with limited offline data. Our empirical results on two public stock datasets show that MetaTrader outperforms existing methods, including both RL-based approaches and traditional stock prediction models.
BibTeX
@misc{yuan2025your,
  author = {Haochen Yuan and Minting Pan and Yunbo Wang and Siyu Gao and Philip S. Yu and Xiaokang Yang},
  title = {{Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization}},
  year = {2025},
  eprint = {2505.12759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.12759},
}
Figure