TL;DR
CATF (Context-Aware Time-Series Forecasting) is a modular framework that uses a "Manager-Worker" architecture to handle complex, context-dependent data. By employing a Transformer-based Manager to "hard route" inputs to specialized Worker models, it achieves state-of-the-art accuracy and high interpretability while keeping inference costs low.
What Problem Does This Paper Address?
Time-series forecasting often depends heavily on external context—think of how bike-sharing demand shifts dramatically between a rainy Monday commute and a sunny Sunday afternoon. Standard single-model approaches struggle to capture these distinct "regimes" simultaneously, often settling for a mediocre average.
While existing solutions like Ensemble Learning or Mixture-of-Experts (MoE) exist, they come with baggage. Ensembles require manual data partitioning and are computationally expensive at inference time. Traditional MoEs use "soft routing," where multiple experts are activated and their outputs weighted, which is both computationally heavy and difficult to interpret. The authors of CATF set out to create a system that allows for automatic model specialization without manual partitioning, ensuring that only one specialized model is active for any given forecast.
Methods
The core of CATF is the Manager-Worker (MW) architecture.
- The Manager ($\mathcal{M}$): A Transformer-based classifier that observes the historical data ($X$) and contextual variables ($C$). Its job is to predict which Worker is best suited for the current input.
- The Workers ($\mathcal{W}_1, \dots, \mathcal{W}_K$): These are the actual forecasting models. CATF is model-agnostic, meaning any standard forecaster (like iTransformer, TimesNet, or FEDformer) can serve as a Worker.
- Hard Routing: Unlike soft-routing MoEs, CATF uses "hard routing." The Manager selects exactly one Worker ($k^* = \arg \max \hat{P}_i$) to perform the forecast. This happens during both training and inference.
The training process is an iterative "talent scouting" loop. To prevent "expert collapse" (where one worker gets all the data and the others starve), the authors introduce a training regularization term. This "underused-worker bonus" encourages the Manager to explore different Workers early in training before they specialize.
Datasets
The authors evaluated CATF on five real-world, context-rich datasets where exogenous variables (like weather and calendar info) strongly drive the target:
- Weather: Predicting $CO_2$ concentration based on meteorological data.
- PM2.5-LA & PM2.5-RV: Air quality forecasting in Los Angeles and Riverside.
- Bike Share: Hourly bike rental counts in Washington, D.C. (highly dependent on calendar/weather).
- Appliances: Energy use in a low-energy house.
Notably, they avoided standard benchmarks like ETT or Electricity because those datasets lack the diverse contextual drivers necessary to test context-aware specialization.
Key Insights and Ideas
- Economic Specialization: The framework is inspired by the principle that agents maximize collective productivity by focusing on tasks they are best suited for. Over time, Workers naturally specialize in specific regimes (e.g., one Worker becomes the "weekend expert").
- Interpretability by Design: Because only one Worker produces each forecast, the system is "transparent." By inspecting the Manager's routing decisions, researchers can see exactly which contextual variables (like "Working Day") triggered a specific model selection.
- Stabilizing Unstable Baselines: Interestingly, CATF was found to stabilize training for older models (like FEDformer) on noisy datasets. By partitioning the data, each Worker faces a simpler, more consistent optimization task.
- Beating the Giants: The authors show that a few small, specialized models can often outperform massive, zero-shot foundation models like Time-MoE, which are far more expensive to run.
Key Results
- Broad Improvement: CATF improved performance over standalone baselines in 162 out of 200 experimental settings (81%).
- State-of-the-Art Performance: The best CATF-enhanced model outperformed the best standalone baseline in 19 of 20 settings.
- Foundation Model Comparison: CATF outperformed the Time-MoE-50M foundation model in 15 of 20 settings, despite being significantly smaller.
- Case Study Success: On the Bike Share dataset, the Manager automatically learned to route inputs based on the day of the week and "working day" status, perfectly aligning with human domain knowledge without any manual labels.
Limitations and Future Work
- Training Overhead: While inference is efficient (only one Worker runs), training is more intensive because the Manager and all $K$ Workers must be trained jointly.
- Manual $K$ Selection: Currently, the number of Workers ($K$) must be set by hand. The authors suggest that future work could involve adaptively determining the number of latent context regimes in the data.
- Heterogeneous Workers: While CATF supports mixing different model architectures (e.g., a CNN worker and a Transformer worker), the authors found this didn't consistently improve results and requires more sophisticated regularization to prevent the "strongest" architecture from dominating.
Summary updated