Introduction
Reinforcement learning (RL) has shown promise in financial markets, yet it frequently fails in real-world deployment. The primary hurdle is the generalization-optimality dilemma: agents trained on historical data often overfit to specific past market conditions, failing to generalize when the underlying distribution shifts. Standard offline RL also suffers from value overestimation, where agents become overly optimistic about actions they haven't explored, leading to risky financial decisions.
To address these challenges, this paper introduces MetaTrader, a "partial-offline" RL framework. MetaTrader utilizes bilevel optimization and synthetic data transformations to train agents capable of navigating unseen, out-of-distribution (OOD) market scenarios.
Methodology
MetaTrader reframes portfolio optimization through several innovative technical components:
Partial-Offline RL Formulation
The authors decouple the environment's state into two distinct branches:
- Market States ($h_t$): Global indicators like stock prices and trends. These are treated as action-free and fixed to the offline dataset, as an individual trader's actions do not impact the global market.
- Balance States ($z_t$): The agent's specific holdings (cash and stock). These are action-dependent. This decoupling allows the agent to safely explore various portfolio allocations over fixed historical market data without needing to model the complex dynamics of the global market itself.
Market Data Transformations
To prevent memorization and simulate OOD conditions, MetaTrader generates synthetic data using three primary transformations:
- Unexpected Events ($F_1$): Inverting the growth rates of top-performing stocks to simulate sudden market crashes.
- Chronological Reversal ($F_2$): Reversing trends to simulate long-term shifts in market regime.
- Downsampling ($F_3$): Capturing multi-scale dynamics by reducing data frequency.
Bilevel Policy Learning and Conservative TD
MetaTrader employs a meta-learning loop. The inner loop trains the policy to maximize profits on standard historical data. The outer loop evaluates and updates parameters against the transformed OOD data, acting as a powerful regularizer that prioritizes safe, generalizable strategies.
Furthermore, the framework introduces Transformation-based Temporal Difference (TD) Learning. To combat value overestimation, the agent computes next-step Q-values for both original and transformed data, using the minimum (worst-case) value as its target. This conservative approach proves more efficient and effective than traditional ensemble-based RL.
Experimental Evaluation
The framework was evaluated on two major public stock datasets: CSI-300 (Chinese market) and NASDAQ-100 (US tech stocks), covering the period from 2011 to 2022.
Key Results
MetaTrader significantly outperformed traditional forecasting models (e.g., FactorVAE) and state-of-the-art RL baselines (e.g., StockFormer, FinRL):
- CSI-300 Performance: MetaTrader achieved a Cumulative Return (CR) of 1.44, a 50% improvement over FactorVAE (0.96 CR) and a 16.1% improvement over StockFormer (1.24 CR).
- Online Adaptation: When fine-tuned on streaming test data, MetaTrader's performance on the NASDAQ-100 improved cumulative returns by over 25% compared to StockFormer (1.58 vs 1.26) and increased the Sharpe ratio by approximately 43%.
- Ablation Study: The custom worst-case TD learning module alone contributed a 9.5% boost in cumulative returns compared to standard SAC TD learning.
Discussion and Future Work
MetaTrader demonstrates that data diversity, achieved through targeted transformations, can be more effective than model diversity in conservative RL. By separating uncontrollable market states from controllable balance states, the framework bypasses the difficulty of state estimation in offline RL.
However, challenges remain. RL-based methods like MetaTrader exhibit higher training variance compared to supervised models. Future research will focus on stabilizing training through better initialization and extending the framework to high-frequency trading scenarios. Additionally, the "partial-offline" MDP formulation holds potential for other non-stationary domains, such as autonomous driving.
Summary updated