Ablation studies in machine learning usually answer "does this part help?" with one number. Researchers delete or replace a module, retrain, and compare a single averaged error. For long-range time-series forecasting, that number averages the error over a forecast that can run 720 steps into the future. Phoneticians faced the same problem decades ago with tongue shapes and vowel formants (the resonant frequencies that identify a vowel). Their answer was smoothing spline analysis of variance (SSANOVA): fit smooth curves to whole trajectories, split the variation into a smooth effect per experimental factor plus smooth interaction effects, and draw confidence bands that show where along the curve two conditions really differ. This note proposes treating the forecast horizon the way phoneticians treat time within a vowel. We would run a full factorial ablation grid, every combination of components switched on or off, with several random seeds. Each configuration's error curve over the horizon would then be decomposed into one smooth effect curve per component and per pair of components, with bands that hold simultaneously across the whole horizon.
The motivating case is the recent finding that large language model (LLM) backbones add little to time-series forecasters. Tan et al. replaced the LLM inside OneFitsAll, Time-LLM, and CALF with nothing, a single attention layer, or a random transformer block, and found the simpler variants matched or beat the originals "in 26 out of 26 test cases for Time-LLM and 19 out of 26 for OneFitsAll." They credited patching, meaning cutting the input series into short segments that are treated as tokens. Zheng et al. reached a similar verdict and traced it to a shallow "pseudo-alignment" between time-series and text embeddings. Both papers judge components by mean squared error (MSE) and mean absolute error averaged over the horizon. Neither can tell an LLM that is useless everywhere from one that helps over the first day and hurts over the last week, with the two effects cancelling in the average. That distinction matters for practitioners choosing a model for a specific look-ahead window, and it matters for the scientific claim that pretrained language weights carry no useful structure for forecasting.
Core Claim
Our claim is that component contributions in long-horizon forecasters depend on the horizon, and in particular that the effect of the backbone and its interaction with patching change in size, and possibly in sign, across the horizon. A two-factor-plus SSANOVA fit to per-step error curves would detect this structure with calibrated simultaneous inference, where horizon-averaged ablation tables cannot. The method is not new; SSANOVA is old. What is new is the object of analysis: the per-step error trajectory from a factorial ablation grid, with seeds and test windows modeled as random curves. To our knowledge from this literature, nobody has studied that object.
Phonetics supplies both the precedent and the warning. McAuliffe and Babel used SSANOVA on 2,564 Buckeye Corpus vowels and found that word frequency centralizes the trajectories of back vowels, a dynamic effect that a midpoint-only regression obscured. That is the motivating evidence that scalar summaries hide localized effects. They also flagged speaker-level confounds, and Gili Fivela et al. showed that SSANOVA comparisons must account for overall shape and position before attributing a difference to the factor of interest. These works constrain the design. The forecasting analog of a speaker is the test window and the training seed, and both must enter the model as random curves rather than as independent replicates.
Research Program
For each forecaster and dataset, we would train every cell of a factorial grid crossing four factors. The first is backbone, with four levels: pretrained LLM, randomly initialized LLM, single attention layer, and none. The second is patching, on or off. The third is input normalization, on or off. The fourth is seed, with five per cell. We would use OneFitsAll as the main testbed because its GPT-2 backbone is cheap enough for a full grid. Time-LLM would be run only on the ETTh1 and Weather cells, because Tan et al. report 3,003 training minutes for Time-LLM on Weather, so a full Time-LLM grid is not affordable.
The response is the log squared error at horizon step $t \in \{1,\dots,720\}$, measured relative to a repeat-last-value forecast so that its scale is comparable across datasets. For configuration $m$, seed $s$, and test window $w$, the model is
$$y_{msw}(t) = \mu(t) + \sum_{c} \alpha_c(t) + \sum_{c<c'} \alpha_{cc'}(t) + b_s(t) + b_w(t) + \varepsilon_{msw}(t).$$
Here $\mu(t)$ is the reference error curve, $\alpha_c(t)$ is the smooth main effect of component level $c$, and $\alpha_{cc'}(t)$ is a pairwise interaction curve. The terms $b_s$ and $b_w$ are smooth random curves for seed and window. All smooth terms carry the usual roughness penalty $\int f''(t)^2\,dt$, and smoothing parameters are chosen by generalized cross-validation.
Overlapping test windows are correlated. We would therefore subsample non-overlapping windows for inference and use all windows only for point estimates. Grids with hundreds of windows times 720 steps exceed naive spline cost, so we would use the rounding-parameter approximation of Helwig and Ma and the large-sample smoothing-parameter selection of the asympirical approach. Bands are made simultaneous by calibrating their width with a sup-statistic over $t$ computed from seed-level resampling. Without that step, scanning 720 points would produce spurious regions of significance.
The pairwise terms are where this work meets the interaction problem in component analysis. Wang et al. found backup heads that take over only when primary heads are removed, which shows that one-at-a-time ablation misreads redundant parts. Fumagalli et al. formalize this by splitting attributions into pure, partial (Shapley-style), and full effects under a functional ANOVA, but only for scalar outputs and tabular or token features. Our factorial grid is a complete game over components at every horizon step, so the same pure, partial, and full quantities become curves in $t$. Because components are retrained when replaced rather than zeroed inside a frozen network, the out-of-distribution "spoofing" artifact identified by Optimal ablation for interpretability does not arise. We note this as a deliberate design choice, not a solved problem: our method speaks to retrained ablations only.
Baselines
The naive reference point is the protocol of Tan et al. and Zheng et al.: horizon-averaged MSE and mean absolute error per configuration, reported at horizons 96, 192, 336, and 720. It is the strongest existing method because it is what the field uses to conclude that LLMs are unnecessary. We would recompute it from the same trained grid, so data, compute, and tuning are identical by construction. The file gives the count of matched-or-exceeded cases quoted above, but it contains no per-dataset MSE values, so no reported result was found on Lacuna for ETT/Weather MSE of the ablated OneFitsAll variants. We would reproduce those numbers ourselves and confirm they match the published direction before any further analysis. Our method beats this baseline if, on at least some datasets, it finds a region of the horizon where a component's effect is significant under simultaneous bands while the horizon-averaged difference is not significant, and if that region replicates on held-out seeds.
The strongest estimation competitor is a per-step block bootstrap over test windows and seeds, with sup-t simultaneous bands on the raw, unsmoothed difference curves. It needs no spline machinery. We would give it the same seeds and windows, and tune its block length on the same validation split used for SSANOVA's smoothing parameters. It is beaten if SSANOVA achieves equal error control with narrower bands, meaning higher detection power for planted effects at five seeds.
A third baseline is scalar Shapley and interaction attribution over the factorial grid in the style of Fumagalli et al., computed on horizon-averaged MSE.
Two ablations isolate our key ingredients. Setting the penalty to zero gives pointwise functional ANOVA and tests whether smoothing contributes. Dropping the window and seed random curves tests whether modeling dependence is necessary for calibration.
Why not just…?
Why not just report MSE at each horizon step and eyeball the curves? Raw per-step error curves from a few seeds are noisy enough that visual differences routinely appear and vanish. Without simultaneous bands, scanning 720 steps guarantees false localized "effects." The A/A experiment below quantifies how often eyeballing, or uncorrected pointwise tests, flags a difference between two seeds of the identical configuration.
Why not just use a block bootstrap with simultaneous bands and skip splines? This is a real threat, and we treat it as a co-equal method rather than a strawman. The case for smoothing is that it pools neighboring horizon steps and so should give narrower bands at the small seed counts that expensive models allow. If the planted-effect experiments show no power gain at five seeds, we would adopt the bootstrap and present the contribution as the factorial, horizon-resolved decomposition itself, independent of the estimator.
Why not just compute Shapley values over the ablation grid on aggregate MSE? That handles interactions but not location. An effect that helps early and hurts late averages to zero, which is exactly the case McAuliffe and Babel's midpoint analysis missed for vowels. The planted sign-flip test directly measures whether scalar Shapley misses what interaction curves catch.
Proposed Experiments
The first decisive check is calibration. We would fit the full model to an A/A grid in which two "levels" are the same configuration trained with disjoint seeds, on ETTh1, ETTm1, and Weather. The fraction of the horizon declared significant should be near zero, and the family-wise false-positive rate should stay at or below the nominal 5%. We would run this with and without random curves to show that ignoring window dependence breaks calibration.
The second is power under planted structure. Using predictions from real trained models, we would construct synthetic configuration pairs whose error differs by a known curve: a gain over steps 1–48 and an equal-area loss over steps 500–720, giving a zero aggregate difference. We would measure detection rate and localization error for SSANOVA, the bootstrap, pointwise functional ANOVA, and scalar Shapley, at three and five seeds.
The third is the substantive test on ETTh1, ETTh2, ETTm1, ETTm2, Weather, ECL, and Traffic with OneFitsAll, and on the two Time-LLM cells. It is falsifiable in both directions. If backbone main-effect bands contain zero everywhere and the backbone-by-patching interaction is flat, we would have strengthened the null result of Tan et al. to a horizon-uniform statement, which is itself worth publishing. If instead the pretrained-versus-random backbone curve departs from zero over a contiguous region that replicates when the seed pool is split in half, the "LLMs are useless" conclusion must be qualified by horizon.
We also expect the patching main effect to grow with $t$, since patches summarize local trend. If it does not, the attribution of gains to patching needs revisiting at the step level.
References
- Are Language Models Actually Useful for Time Series Forecasting?
- Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis: The Impact of Modality Alignment
- Predictability affects vowel dispersion and dynamics in the Buckeye corpus
- Coarticulation across prosodic domains in Italian: an ultrasound investigation
- Smoothing spline ANOVA for super-large samples: scalable computation via rounding parameters
- An Asympirical Smoothing Parameters Selection Approach for Smoothing Spline ANOVA Models in Large Samples
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
- Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory
- Optimal ablation for interpretability