Letting the Forecaster Verify Its Own Context: Counterfactual Group Advantages for Text-Aided Forecasting

Forecasting models that read text alongside numbers ("a plant outage is expected on day two") often do no better than models that read the numbers alone. One reason is that the training text is noisy, so the model learns that ignoring it is safe. The best current fix filters the synthetic training text offline. It asks a frontier language model to forecast with and without each text snippet and keeps a snippet only if it lowers the forecast error. That check is expensive and tied to a single judge. In the original work it was applied only to the 904 test windows, and the 7.4 million training windows were never verified.

This note proposes moving that check inside reinforcement learning. For each training window, the forecaster samples a group of future trajectories with the text visible and another group with the text masked. Every trajectory is scored against the true future, and each trajectory's advantage is computed against the pooled group. A text-conditioned trajectory is therefore rewarded only when it beats the model's own text-blind forecasts on the same window. When the text is uninformative, the two halves are statistically interchangeable and the text-specific gradient averages to zero, so the window is effectively filtered. When the text helps, the model is pushed to use it. The data filter and the fix for the modality gap become a single training signal, and no external judge is involved.

Core Claim

The offline criterion "keep text only if it improves CRPS" can be rewritten as a critic-free, group-relative advantage. Here CRPS (Continuous Ranked Probability Score) is a proper scoring rule for probabilistic forecasts: it rewards a forecast distribution for being both accurate and appropriately spread. Group Relative Policy Optimization (GRPO) is a reinforcement learning method that scores each sampled output relative to the other outputs drawn for the same input, instead of against a learned value estimate.

We claim three things. First, this counterfactual advantage can be computed for the whole unverified training corpus at the cost of sampling alone. Second, it produces a forecaster that uses informative text more and misleading text less than supervised training on the same data. Third, most of the gain comes specifically from the pooled comparison of text-visible against text-masked samples, not from reinforcement learning or CRPS rewards alone.

Where This Sits

The immediate motivation is Overcoming the Modality Gap in Context-Aided Forecasting. That paper built CAF-7M from Llama-3-70B narratives and verified test windows by GPT-5.2 CRPS gain. It trained DoubleCast, which couples Chronos to a Qwen text encoder via cross-attention, by supervised fine-tuning only. It also names verifier bias, the unverified training split and weak temporal grounding as open problems. We reuse its model, data and verification criterion, but turn the criterion into an on-policy reward.

Consistency-based filtering motivates the "agreement between two routes" logic. TableLLM keeps table-reasoning examples only when chain-of-thought and executed code agree. Promptagator keeps synthetic queries only when they retrieve their source document. Both filter offline and then fine-tune. Our two routes are "with context" and "without context," and the comparison happens inside training.

Consensus Group Relative Policy Optimization is the closest methodological relative. It rewards each text rollout for agreeing with its peers, without references. Our reward instead has access to ground-truth futures. It deliberately rewards disagreement between peers, because CRPS pays for spread. The comparison we care about is across conditions, not within one condition.

Existing reinforcement learning work on time series computes rewards against fixed targets and never tests whether a second modality earned its keep. FinZero renders series as chart images and uses uncertainty-adjusted group advantages. FAKER-Air applies GRPO with a class-weighted air-quality reward.

Two papers constrain the design. MO-GRPO and Constrained GRPO show that summing heterogeneous reward terms before group normalization lets the highest-variance term dominate. We therefore use a single proper-scoring reward. Any auxiliary term, such as output validity, is normalized separately.

Research Program

The policy is a DoubleCast-style model: Chronos quantized-value tokens plus a cross-attending text encoder. We train it by the paper's recipe, or use a released checkpoint if one exists. Because Chronos samples forecast trajectories token by token over value bins, GRPO applies directly.

For a window with history $h$, context $c$ and true future $y$, we sample $G$ trajectories with $c$ visible and $G$ with $c$ replaced by an empty context. The reward for trajectory $x_i$ is its exact share of the ensemble CRPS gradient, computed within its own condition half:

$$r_i = -\sum_{t=1}^{H} \frac{1}{s}\Big(|x_{i,t} - y_t| - \frac{1}{G-1}\sum_{j \neq i} |x_{i,t} - x_{j,t}|\Big).$$

Here $H$ is the horizon, $s$ is the mean absolute value of the history (for scale-free rewards), and $j$ ranges over the other samples in the same condition. The unit coefficient on the spread term comes from differentiating $\mathbb{E}|X-y| - \tfrac12\mathbb{E}|X-X'|$ with respect to the policy. Because both $X$ and $X'$ are drawn from it, the pairwise term contributes twice. Rewards are therefore unbiased per-sample credit for the ensemble's CRPS. They are also continuous, which largely avoids the zero-variance failure of GRPO with binary rewards.

Advantages are standardized over the pooled $2G$ samples. Pooling is the key ingredient. If each condition were normalized separately, the text-visible samples would only compete with each other, and the model could keep ignoring text.

Two by-products are measured, not assumed. The first is a per-window context-gain estimate $g$: the mean reward of the text-visible half minus the mean reward of the text-masked half. Tracked over training, it gives an implicit, judge-free verification score for all training windows. The second is a text-sensitivity profile, which records how forecasts change when the context is swapped, shuffled or time-shifted. Training runs on a subsample of CAF-7M chosen to match the baselines' compute. Group size is $G=16$ per condition, with a KL penalty to the starting checkpoint.

Baselines

The naive reference point is unimodal Chronos with no text. It answers whether text helps at all.

The strongest existing method is DoubleCast trained by supervised fine-tuning on CAF-7M. The file reports a 71% win rate against Chronos on the verified 904-window CAF-7M test split (Overcoming the Modality Gap in Context-Aided Forecasting). We evaluate on that same split. Beating DoubleCast means a statistically significant higher win rate against Chronos and lower mean CRPS than DoubleCast on the same 904 windows. We also require no degradation on numbers-only data.

The most dangerous competitor is offline self-filtering plus supervised fine-tuning. Here the starting DoubleCast checkpoint itself computes the text-versus-no-text CRPS gain on the training windows, which replaces GPT-5.2 with a cheap open verifier. Only windows with positive gain are kept, and fine-tuning continues on them. It uses the same criterion as our method, without reinforcement learning. We give it the same total sampling-plus-training GPU budget and the same tuning budget for its threshold and learning rate.

A second strong competitor is a differentiable counterfactual loss. We add to the supervised objective a hinge penalty, applied whenever the per-step marginal CRPS computed from the bin probabilities with text exceeds the stop-gradient CRPS without text. This loss is trained on the same data and budget.

Three ablations isolate the proposal's key ingredient. The first is plain GRPO with the same CRPS reward and no masked half, which tests whether any benefit is simply reinforcement learning on CRPS. The second keeps the masked half but normalizes each condition separately, which tests whether the pooled comparison matters. The third is GRPO run only on windows retained by the offline self-filter, which tests whether in-loop filtering adds anything beyond pre-filtering.

Why not just…?

Why not just filter the training set offline with the model itself and fine-tune? This is the most serious threat. It captures the same "does text help" criterion cheaply. Its weakness is circularity. A checkpoint that already ignores text will measure near-zero gain everywhere and discard informative windows along with noise. Promptagator already shows that strict consistency filters can remove much of a niche dataset. The in-loop version re-estimates gain as the model learns to read text. The matched-compute comparison above settles this. If offline self-filtering ties our method, we will report that the value lies in the criterion, not in reinforcement learning.

Why not just run GRPO with a CRPS reward? Ground-truth rewards alone do not require the model to use text. A policy can improve CRPS entirely from history. The unpaired-GRPO ablation measures this directly, and we expect it to improve numeric CRPS without raising text sensitivity. This expectation is based on DoubleCast's finding that the multimodal model behaves almost identically to Chronos on numbers-only data.

Why not just add a differentiable counterfactual loss and skip reinforcement learning? This is also a real threat, because per-step CRPS over quantized bins is differentiable. Its limitation is that it scores marginal distributions step by step under teacher forcing. The GRPO reward scores whole sampled trajectories, so it penalizes paths that drift, the exposure-bias problem FAKER-Air addressed with reinforcement learning. Whether that matters at CAF horizons is an empirical question, and the head-to-head comparison answers it.

Why not just have GPT-5.2 verify all 7.4 million windows? The source paper could not afford to verify beyond the test split. It also flags single-judge bias. Our method requires no external calls.

Proposed Experiments

The primary evaluation uses the 904-window verified test split of CAF-7M, reporting win rate against Chronos and mean CRPS against all baselines. For real-world transfer we use the CGTSF benchmark. No reported result for DoubleCast on CGTSF was found on Lacuna beyond the statement that it beat Chronos and ChatTime in almost every category, so we will rerun DoubleCast ourselves. For numeric-skill preservation we use GIFT-Eval with text removed. No reported number was found on Lacuna there either; the paper states only near-parity with Chronos.

We run three decisive diagnostics. The first is a misleading-context test: each window's context is swapped with another window's. A better model should degrade less than DoubleCast on swapped context while gaining more on true context, and the pooled-normalization ablation should lose this property.

The second is a temporal-grounding test. On the subset of contexts that mention a timed event, identified by pattern-matching on day and step references, we shift the stated time. We measure whether the forecast's change point moves accordingly. This targets the grounding failure the source paper reports.

The third checks the implicit filter. On the 904 test windows, all of which GPT-5.2 judged positive, we measure how often the learned gain estimate $g$ is positive. On a random sample of training windows, we compare $g$ to the offline self-filter's keep-or-drop decisions. Low agreement combined with a better final model would indicate that in-loop verification recovers windows the offline filter wrongly drops.

The claim fails if the matched-compute offline self-filter or the differentiable counterfactual loss equals our win rate and misleading-context robustness. It also fails if pooled normalization makes no difference relative to per-condition normalization.