TL;DR
FlexCast is a new framework for time series foundation models that replaces slow autoregressive (AR) decoding with a flexible "mask-and-predict" approach. By disentangling training distribution from inference strategy, the authors demonstrate that the success of AR models stems from diverse training contexts rather than sequential decoding, allowing FlexCast to achieve state-of-the-art accuracy with significantly lower latency.
What Problem Does This Paper Address?
In the world of Time Series Foundation Models (TSFMs), there is a divide. On one side, Autoregressive (AR) models (like Chronos or TimesFM) are highly flexible and generalize well because they learn to predict the next "token" based on varying history lengths. However, they are slow at inference because they must generate the future one step at a time. On the other side, Non-Autoregressive models are fast (one-step prediction) but often "rigid," trained on fixed observation-to-prediction windows that don't adapt well to new tasks.
The authors ask a diagnostic question: What actually makes AR models work? Is it the slow, step-by-step inference, or is it simply that they are exposed to a diverse distribution of context lengths during training? FlexCast aims to answer this by creating a framework that offers the training flexibility of AR models without the inference-time bottleneck.
Methods
FlexCast reformulates forecasting as a variable-length masked imputation task over time series patches.
- Mask-and-Predict Strategy: During pretraining, the model receives a sequence where a contiguous span of patches is masked. The model must reconstruct these patches. By dynamically varying the position and length of this mask, the model is exposed to a massive variety of "context-target" configurations—mimicking the diversity of AR training without requiring sequential steps.
- Unified Interface: Because the model learns via masking, the same pretrained weights can be used for different inference schedules:
- One-step: Predict the entire horizon in a single pass (fastest).
- Confidence-ordered: Iteratively reveal the most "certain" patches first (using quantile widths as a confidence measure).
- Autoregressive: Emulate traditional left-to-right rollout.
- Backbone: The architecture uses a standard Transformer encoder with Rotary Positional Embeddings (RoPE) to handle varying sequence lengths and a Quantile Linear Head to output probabilistic forecasts rather than just single point estimates.
Datasets
- Pretraining: The model was trained on the BLAST (Balanced Sampling Time Series) corpus, which contains roughly 19.8 million training sequences (~81 billion time points) across diverse domains like climate, energy, and finance.
- Evaluation: Zero-shot performance was tested on standard benchmarks: ETTh1, ETTh2, ETTm1, ETTm2, Weather, and GlobalTemp.
Key Insights and Ideas
- The "Aha" Moment: The primary benefit of the AR paradigm is flexible training, not sequential inference. Once a model has learned from diverse context-target relationships, forcing it to decode left-to-right at inference time often yields no accuracy gain and sometimes even increases error due to error accumulation.
- Contiguous Masking is Key: Unlike natural language processing where random token masking is standard, time series models perform better when masking contiguous spans. This encourages the model to capture long-term temporal trends rather than just interpolating from immediate neighbors.
- Uncertainty as a Guide: By predicting quantiles, FlexCast can estimate its own confidence. In iterative modes, it can "fill in" the parts of the future it is most sure about first, then use those as additional context to refine the harder-to-predict segments.
Key Results
- Accuracy: FlexCast achieved competitive or superior zero-shot accuracy compared to heavyweights like PatchTST-FM, Sundial, and Chronos. In the BLAST-retrained category, it secured the most "wins" across MSE and MAE metrics.
- Efficiency: Because it can predict the entire horizon in one pass, FlexCast is significantly faster than AR models. As the prediction horizon grows, the latency gap widens, making FlexCast much more scalable for long-term forecasting.
- Ablation Findings: Ablations confirmed that "Flexible Context" (training on many history lengths) was the single most important factor for generalization.
Limitations and Future Work
- Training Efficiency: The current implementation uses padding for variable-length sequences within a batch, which wastes some computation. The authors suggest "packed" training (concatenating multiple short sequences) as a future optimization.
- Adaptive Schedules: While the paper explores fixed iterative rounds, future work could involve adaptive reveal schedules that decide how many iterations are needed based on the specific difficulty of the data.
- Scaling: The study focused on the BLAST corpus. Testing FlexCast on trillion-scale datasets (like those used for the largest industry TSFMs) is a logical next step to see if these insights hold at extreme scales.
Summary updated