Adaptive Meta-Learning Probabilistic Inference Framework for Long Sequence Prediction

Jianping Zhu, Xin Guo, Yang Chen, Yao Yang, Wenbo Li, Bo Jin, Fei WuAAAI 2024 · DOI

Long-sequence prediction (LSP) is a critical task in domains ranging from financial forecasting to weather modeling. However, the ability to look far into the future is often hampered by two major hurdles: complex, entangled data patterns (trends mixed with seasonal noise) and "domain shift"—the tendency for data distributions to change over time or across different entities.

In a paper presented at AAAI-2024, researchers introduced AMPIF (Adaptive Meta-Learning Probabilistic Inference Framework), a novel approach designed to make long-term forecasting more robust and adaptable by decomposing data into seasonal and trend components and adaptively clustering them into tasks.

Methodology

The AMPIF framework operates through four sophisticated stages to address the limitations of traditional models:

  1. Frequency Domain Decomposition: Instead of using simple moving averages, AMPIF uses Fast Fourier Transforms (FFT) to decouple sequences into seasonal and trend components. It specifically filters out high-frequency noise to ensure the seasonal component is clean.
  2. Adaptive Task Construction: Rather than assigning tasks based on prior knowledge, AMPIF uses a clustering-matching strategy. It groups similar seasonal and trend patterns from a "support set" into clusters. New "query" data is then matched to these clusters, ensuring the model learns from the most relevant historical patterns.
  3. Global Representation Learning (ST-GRL): To handle domain shift, the framework uses a Variational Autoencoder (VAE) structure to learn global representations. Mutual information constraints are added to ensure the learned features capture the relationship between both the input sequence and the future labels, preventing "posterior collapse."
  4. Dual-Stream Amortized Network (ST-DAN): This meta-engine consists of two parallel streams (seasonality and trends). It uses the support set to quickly generate task-specific parameters through a single forward pass, allowing for rapid generalization without expensive second-order gradient calculations.
The architecture of the AMPIF framework, illustrating the decomposition, clustering, and dual-stream network.

Experimental Evaluation

The authors evaluated AMPIF on three distinct types of data: historical A-share stock data (2017–2022), a Cross-Market dataset of 22 global market indices, and the standard Electricity Transformer Temperature (ETT) benchmark.

Key Results and Insights

  • Financial Markets: On the A-share dataset, AMPIF significantly outperformed state-of-the-art models like Autoformer and LaST. In terms of Mean Squared Error (MSE), AMPIF often achieved results 30-50% better than the next best baseline.
  • Standard Benchmarks: On the ETT datasets, AMPIF reduced the average MSE by 15.8% and MAE by 8.5% compared to the popular Autoformer model.
  • Robustness in Scarcity: One of the most striking findings was AMPIF's performance with limited data. While standard models' accuracy plummeted when trained on only 1% of the data, AMPIF remained remarkably stable.
Comparison of MSE performance across different training data ratios, highlighting AMPIF's stability in low-data regimes.

Discussion and Conclusion

The research highlights that decoupling seasonal and trend terms is key to improving generalization. By clustering tasks based on actual data fluctuations rather than arbitrary categories, the model avoids the bias of human-defined task boundaries. Furthermore, by explicitly maximizing mutual information, the model creates a more effective global view of data dynamics.

While AMPIF shows great promise, future work could address hyperparameter sensitivity and explore more diverse backbone architectures. The framework's efficiency, achieved through amortized parameter generation, makes it a practical candidate for real-world deployment in volatile environments.

Summary updated

BibTeX
@inproceedings{zhu2024adaptive,
  author = {Jianping Zhu and Xin Guo and Yang Chen and Yao Yang and Wenbo Li and Bo Jin and Fei Wu},
  title = {{Adaptive Meta-Learning Probabilistic Inference Framework for Long Sequence Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i15.29661},
}
Figure