TimeSQL: Improving Multivariate Time Series Forecasting with Multi-Scale Patching and Smooth Quadratic Loss

Site Mo, Haoxin Wang, Bixiong Li, Songhai Fan, Yuankai Wu, Xianggen LiuarXiv 2023 · DOI · arXiv

Abstract

Accurate multivariate time series forecasting is often hindered by the inherent noise of real-world data and the presence of complex patterns across multiple temporal scales. This paper introduces TimeSQL, a framework that enhances forecasting robustness through two key innovations: multi-scale patching and a novel Smooth Quadratic Loss (SQL). By dampening the influence of outliers while capturing both local and global temporal dependencies, TimeSQL achieves state-of-the-art results across eight major benchmarks, providing a versatile "plug-and-play" enhancement for existing neural architectures.

Introduction

Predicting the future states of complex systems—such as power grids, traffic networks, or disease outbreaks—is a fundamental challenge in data science. Real-world time series data typically present two major hurdles. First, temporal complexity requires models to simultaneously perceive short-term volatility and long-term seasonal trends. Second, the outlier problem makes standard training objectives like Mean Squared Error (MSE) problematic; because MSE gradients increase linearly with error size, models often overfit to noisy spikes at the expense of the underlying signal. TimeSQL addresses these issues by rethinking both data representation and the optimization objective.

Methodology

Multi-Scale Patching

Rather than processing raw individual data points, TimeSQL utilizes a patching mechanism inspired by Vision Transformers. It segments the time series into patches of varying lengths simultaneously. Short patches allow the model to focus on local fluctuations, while longer patches capture broader correlations. These multi-scale representations are processed by standard encoders (e.g., LSTMs, CNNs, or Transformers) and subsequently fused to generate final predictions.

Overview of the TimeSQL framework, illustrating the pipeline from multi-scale patching to encoder processing and the SQL-based parameter update loop.

Smooth Quadratic Loss (SQL)

The core innovation of TimeSQL is the Smooth Quadratic Loss (SQL), a robust objective function derived from the rational quadratic kernel. Unlike MSE, the gradient of the SQL function is non-linear and specifically designed to handle outliers. For standard data points, the gradient provides sufficient signal for learning, but for extreme errors (likely noise), the gradient strength decreases, preventing the model from overreacting to anomalies.

The SQL function is composed of three integrated components:

  1. Rational Quadratic Function (RQF): Provides non-linear noise filtering.
  2. Mean Absolute Error (MAE): Ensures smoothness and handles low-latency information.
  3. Outlier Regularization (OR): Penalizes predictions that deviate excessively from the expected range, further stabilizing training.
Comparison of Loss and Gradient behaviors for RQF, SQL, and MSE. Note how SQL gradients dampen for large errors, whereas MSE gradients continue to grow linearly.

Experimental Evaluation

TimeSQL was evaluated on eight diverse multivariate time series benchmarks, including Weather, Traffic, Electricity, and Influenza-like Illness (ILI) datasets. The framework consistently outperformed previous state-of-the-art (SOTA) models such as PatchTST and DLinear.

  • Performance Gains: TimeSQL reduced the Mean Absolute Error (MAE) by 5.3% on Weather data, 13.4% on the ILI dataset, and 4.1% on the ETTh1 electricity transformer dataset compared to PatchTST.
  • Robustness to Horizon: As prediction horizons increased, TimeSQL exhibited significantly slower error growth than competing models, indicating superior long-term forecasting stability.
  • Versatility: A key finding was the "plug-and-play" nature of the SQL loss. Replacing MSE with SQL in other architectures like Informer or PatchTST yielded immediate performance improvements without requiring structural changes to those models.

Conclusion

TimeSQL represents a significant advancement in noise-aware time series forecasting. By combining multi-scale feature extraction with a mathematically robust loss function, it shifts the focus from architectural scaling to more resilient optimization strategies. While the framework introduces new hyperparameters that require tuning, its consistent success across varied domains establishes it as a powerful tool for real-world multivariate forecasting tasks.

Summary updated

BibTeX
@misc{mo2023timesql,
  author = {Site Mo and Haoxin Wang and Bixiong Li and Songhai Fan and Yuankai Wu and Xianggen Liu},
  title = {{TimeSQL: Improving Multivariate Time Series Forecasting with Multi-Scale Patching and Smooth Quadratic Loss}},
  year = {2023},
  eprint = {2311.11285},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11285},
}
Figure