Ground-Truth Neighborhood Regularization for Reinforcement Learning Post-Training of Time Series Foundation Models

Jianqi Zhang, Xingyu Zhang, Zeen Song, Changwen Zheng, Fanjiang Xu, Wenwen QiangarXiv 2026 · arXiv

TL;DR

Reinforcement learning (RL) post-training can significantly boost Time Series Foundation Models (TSFMs), but it often suffers from "suboptimal collapse"—a vicious cycle where the model's predicted distribution drifts completely away from the ground truth. To solve this, the authors introduce Ground-Truth Neighborhood Regularization (GTN-R), a method that gently guides the model's probability mass toward a window around the ground truth while preserving its vital exploration capabilities.

What Problem Does This Paper Address?

Pretrained Time Series Foundation Models (TSFMs) are powerful, but they often need downstream adaptation to excel at specific tasks. While Reinforcement Learning (RL) post-training has emerged as a superior alternative to standard Supervised Fine-Tuning (SFT), the authors discover a critical flaw: suboptimal collapse.

During RL training, the model's predictive distribution can gradually concentrate outside the ground-truth neighborhood in certain regions.

Figure 1: Illustration of suboptimal collapse during RL post-training of Moirai-small. Each row shows the evolv- ing predictive distribution for a training sample, with the RL method, processed epoch, and MSE reported above each panel. The TSFM used is Moirai-small.As training pro- gresses, distribution within the yellow boxes shifts toward regions far from the ground truth, indicating suboptimal col- lapse. More results are in Appendix “More Visualization.”

This happens because of initial sampling difficulties. If a model initially struggles to sample trajectories near the ground truth, the RL algorithm will simply reward the "best of a bad bunch." This creates a detrimental self-reinforcing feedback loop: the model reinforces low-quality regions, making it even less likely to sample near the ground truth in subsequent iterations.

Simply adding the ground-truth trajectory directly into the RL training pool doesn't work either; it causes the model to lazily mimic the data (behavior cloning), destroying its generalization and exploration capabilities.

Methods

The authors propose Ground-Truth Neighborhood Regularization (GTN-R) to mitigate this collapse. Instead of forcing the model to rigidly copy the ground truth, GTN-R adds two distributional constraints to the standard RL objective:

  1. Neighborhood Probability Mass Constraint ($\mathcal{L}_{mass}$): This maximizes the total probability mass the model assigns to a small window (neighborhood) centered around the ground truth. It uses a logarithmic penalty so that if the model assigns a near-zero probability to the correct region, it gets heavily penalized.
  2. Within-Neighborhood Uniformity Constraint ($\mathcal{L}_{uni}$): To prevent the model from collapsing its entire probability mass onto a single point within that neighborhood (which ruins exploration), this constraint uses KL divergence to push the distribution inside the neighborhood toward a uniform distribution.
Overall framework of GTN-R

GTN-R can be flexibly integrated into various RL post-training frameworks, such as TPO, TimeRFT, or the authors' own TS-GRPO.

Datasets

The method was evaluated on eight widely used, real-world time series benchmarks:

  • ETT (ETTh1, ETTh2, ETTm1, ETTm2)
  • ECL (Electricity Consumption Load)
  • Weather
  • Loop Seattle (Traffic)
  • ENTSO-e Load (Energy)

The datasets were evaluated under both standard and zero-shot forecasting settings. No new benchmarks were introduced.

Key Insights and Ideas

  • The "Best of a Bad Bunch" Trap: Because RL relies on relative rewards within a sampled group, if all samples are poor, a bad sample can still be reinforced. This is the mathematical root cause of suboptimal collapse.
  • Soft Guidance > Hard Imitation: Directly injecting the ground truth into RL updates degrades performance because it turns an exploration-driven RL process into rigid behavior cloning. Soft regularization via neighborhoods keeps the exploration alive.
  • Uniformity Preserves Exploration: Forcing the model to maintain a near-uniform distribution within the correct neighborhood ensures it doesn't prematurely lock into a single trajectory, preserving its ability to handle real-world stochasticity and uncertainty.

Key Results

  • Consistent Performance Gains: GTN-R was tested across multiple TSFM backbones (Moirai-small, Moirai-base, Toto, Moirai-MoEs, UniTS) and RL frameworks. It consistently reduced Mean Squared Error (MSE) and Mean Absolute Error (MAE) across the board. For instance, adding GTN-R to Moirai-small + TS-GRPO on the ETTm1 dataset reduced the MSE from 0.358 to 0.345.
  • Mitigating Collapse While Maintaining Variance: Qualitative evaluations showed that GTN-R successfully arrested suboptimal collapse (keeping the model's distribution centered around the true future values) without causing the sampling variance to plummet.

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Time series forecasting (TSF) plays an important role in a wide range of real-world applications. Recently, time series foundation models (TSFMs), pretrained on large-scale datasets, have demonstrated strong generalization capabilities and emerged as an important paradigm for TSF. Reinforcement learning (RL) post-training has consequently attracted growing attention as a means of further improving their performance on downstream tasks. However, we find that, in certain forecast regions, RL post-training may gradually shift the output distributions of TSFMs away from the ground truth, thereby limiting their performance. We refer to this phenomenon as \textbf{suboptimal collapse}. Our analysis suggests that difficulty in initially sampling high-quality trajectories near the ground truth is an important contributing factor to suboptimal collapse. To address this issue, we propose Ground-Truth Neighborhood Regularization (GTN-R) for RL post-training of TSFMs. GTN-R uses the ground truth as a reference for locating high-quality regions and guides the model's probability mass toward the ground-truth neighborhood. This increases the probability of sampling high-quality trajectories, mitigates suboptimal collapse, and improves performance. Moreover, GTN-R can be flexibly integrated into various RL methods for TSFMs. Extensive experiments show its effectiveness.
BibTeX
@misc{zhang2026ground,
  author = {Jianqi Zhang and Xingyu Zhang and Zeen Song and Changwen Zheng and Fanjiang Xu and Wenwen Qiang},
  title = {{Ground-Truth Neighborhood Regularization for Reinforcement Learning Post-Training of Time Series Foundation Models}},
  year = {2026},
  eprint = {2608.08010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.08010},
}
Figure