TL;DR
Reinforcement learning (RL) post-training can significantly boost Time Series Foundation Models (TSFMs), but it often suffers from "suboptimal collapse"—a vicious cycle where the model's predicted distribution drifts completely away from the ground truth. To solve this, the authors introduce Ground-Truth Neighborhood Regularization (GTN-R), a method that gently guides the model's probability mass toward a window around the ground truth while preserving its vital exploration capabilities.
What Problem Does This Paper Address?
Pretrained Time Series Foundation Models (TSFMs) are powerful, but they often need downstream adaptation to excel at specific tasks. While Reinforcement Learning (RL) post-training has emerged as a superior alternative to standard Supervised Fine-Tuning (SFT), the authors discover a critical flaw: suboptimal collapse.
During RL training, the model's predictive distribution can gradually concentrate outside the ground-truth neighborhood in certain regions.
This happens because of initial sampling difficulties. If a model initially struggles to sample trajectories near the ground truth, the RL algorithm will simply reward the "best of a bad bunch." This creates a detrimental self-reinforcing feedback loop: the model reinforces low-quality regions, making it even less likely to sample near the ground truth in subsequent iterations.
Simply adding the ground-truth trajectory directly into the RL training pool doesn't work either; it causes the model to lazily mimic the data (behavior cloning), destroying its generalization and exploration capabilities.
Methods
The authors propose Ground-Truth Neighborhood Regularization (GTN-R) to mitigate this collapse. Instead of forcing the model to rigidly copy the ground truth, GTN-R adds two distributional constraints to the standard RL objective:
- Neighborhood Probability Mass Constraint ($\mathcal{L}_{mass}$): This maximizes the total probability mass the model assigns to a small window (neighborhood) centered around the ground truth. It uses a logarithmic penalty so that if the model assigns a near-zero probability to the correct region, it gets heavily penalized.
- Within-Neighborhood Uniformity Constraint ($\mathcal{L}_{uni}$): To prevent the model from collapsing its entire probability mass onto a single point within that neighborhood (which ruins exploration), this constraint uses KL divergence to push the distribution inside the neighborhood toward a uniform distribution.
GTN-R can be flexibly integrated into various RL post-training frameworks, such as TPO, TimeRFT, or the authors' own TS-GRPO.
Datasets
The method was evaluated on eight widely used, real-world time series benchmarks:
- ETT (ETTh1, ETTh2, ETTm1, ETTm2)
- ECL (Electricity Consumption Load)
- Weather
- Loop Seattle (Traffic)
- ENTSO-e Load (Energy)
The datasets were evaluated under both standard and zero-shot forecasting settings. No new benchmarks were introduced.
Key Insights and Ideas
- The "Best of a Bad Bunch" Trap: Because RL relies on relative rewards within a sampled group, if all samples are poor, a bad sample can still be reinforced. This is the mathematical root cause of suboptimal collapse.
- Soft Guidance > Hard Imitation: Directly injecting the ground truth into RL updates degrades performance because it turns an exploration-driven RL process into rigid behavior cloning. Soft regularization via neighborhoods keeps the exploration alive.
- Uniformity Preserves Exploration: Forcing the model to maintain a near-uniform distribution within the correct neighborhood ensures it doesn't prematurely lock into a single trajectory, preserving its ability to handle real-world stochasticity and uncertainty.
Key Results
- Consistent Performance Gains: GTN-R was tested across multiple TSFM backbones (Moirai-small, Moirai-base, Toto, Moirai-MoEs, UniTS) and RL frameworks. It consistently reduced Mean Squared Error (MSE) and Mean Absolute Error (MAE) across the board. For instance, adding GTN-R to
Moirai-small + TS-GRPOon the ETTm1 dataset reduced the MSE from0.358to0.345. - Mitigating Collapse While Maintaining Variance: Qualitative evaluations showed that GTN-R successfully arrested suboptimal collapse (keeping the model's distribution centered around the true future values) without causing the sampling variance to plummet.
Limitations and Future Work
Not discussed in the paper.
Summary updated