Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting

Min Chen, Guansong Pang, Wenjun Wang, Cheng Yan

TL;DR

This paper introduces RSTIB-MLP, a method that makes simple Multi-Layer Perceptrons (MLPs) highly robust to noise in spatial-temporal forecasting without sacrificing their computational efficiency. By identifying a "dual noise effect" in time-series data and applying a novel Robust Spatial-Temporal Information Bottleneck (RSTIB) principle, the authors achieve a sweet spot between the blazing speed of MLPs and the noise-resilience of complex Graph Neural Networks.

What Problem Does This Paper Address?

Spatial-temporal forecasting (STF)—used for predicting traffic, weather, or energy consumption—relies heavily on Spatial-Temporal Graph Neural Networks (STGNNs). While STGNNs are great at handling the complex, noisy dynamics of the real world, they are computationally heavy and slow to train.

Recently, researchers have tried replacing STGNNs with simple Multi-Layer Perceptrons (MLPs) to speed things up. However, standard MLPs lack the architectural complexity to filter out noise, leading to severe performance drops when real-world data gets messy.

Flow data with and without noise

The authors identify a specific challenge in STF called the dual noise effect. Because forecasting models use a sliding window approach, a noisy sequence of data might act as the input (history) in one window, but as the target (future prediction) in the next. Traditional robust learning methods assume the target data is clean, which completely fails in this sliding-window scenario where noise corrupts both ends.

Methods

To solve this, the authors propose a new theoretical framework and a practical architecture:

  1. Robust Spatial-Temporal Information Bottleneck (RSTIB): The Information Bottleneck (IB) principle is a classic way to learn robust features by compressing the input to keep only the information necessary to predict the target. However, standard IB assumes a strict Markov chain that ignores noise in the target. RSTIB mathematically lifts this restriction, explicitly reformulating the objective to minimize noisy, irrelevant information coming from both the input and the target.
  2. RSTIB-MLP: This is the practical instantiation of the RSTIB principle using pure MLPs. It uses data reparameterization (injecting controlled Gaussian noise during training) to enforce input, target, and representation regularization, avoiding the need for complex graph structures.
  3. Dynamic Training via Knowledge Distillation (KD): Applying a blanket regularization to all time series is suboptimal because some sensors/nodes are noisier than others. The authors use a pre-trained "teacher" model to calculate a "noise impact indicator" for each time series. This KD module dynamically adjusts the regularization strength: relaxing it for clean data to preserve features, and intensifying it for noisy data to suppress corruption.

Datasets

The authors evaluated their model on a diverse set of benchmarks, artificially injecting varying ratios of Gaussian noise (from 10% up to 90%) to test robustness:

  • Traffic Flow: PEMS04, PEMS07, PEMS08, and LargeST(SD) (a large-scale benchmark).
  • Meteorology: Weather2K-R (predicting vertical visibility).
  • Energy: Electricity (average electricity consumption).

Key Insights and Ideas

  • The Target is Not Ground Truth: The realization that sliding windows cause noise to bleed into the target variable is a crucial paradigm shift. By acknowledging the "dual noise effect," models can be designed to actively distrust and filter the target, rather than blindly fitting to it.
  • IB Doesn't Need Graphs: Previous robust IB methods for spatial-temporal data relied heavily on Graph Neural Networks to prune noisy edges. RSTIB proves that with the right mathematical formulation and reparameterization, you can achieve IB-level robustness using simple, graph-free MLPs.
  • Preventing Feature Collapse: Under heavy noise, standard models suffer from "feature collapse"—their learned embeddings lose diversity and become indistinguishable. The noise-aware KD training regime successfully maintains high feature variance, ensuring the model can still tell different spatial-temporal patterns apart even when the data is highly corrupted.
Feature variance across different noise ratios

Key Results

  • Superior Robustness: Under heavy noise (e.g., 50% noise ratio), RSTIB-MLP consistently outperformed both standard MLPs and complex STGNNs. For example, on the PEMS04 dataset at 50% noise, RSTIB-MLP achieved a Mean Absolute Error (MAE) of 27.16, significantly beating the standard MLP baseline STID (36.22) and the popular graph model GraphWaveNet (40.87).
  • High Efficiency: Despite its robustness, RSTIB-MLP trains much faster than STGNNs. On PEMS04, it required roughly 50 seconds per epoch, compared to well over 100 seconds for advanced STGNNs like DSTAGNN or STG-NCDE. It remains highly competitive with the fastest simple MLP baselines.
  • Missing Data Resilience: In additional tests where data was randomly dropped (up to 30% missing data), RSTIB-MLP maintained lower error rates than competing robust baselines, proving its versatility against different types of data corruption.

Limitations and Future Work

  • Manual Hyperparameter Tuning: The regularization coefficients (Lagrange multipliers) that balance the RSTIB objective are currently fixed and require manual tuning. The authors note that automatically searching for these coefficients with theoretical guarantees is a promising future direction.
  • Slight Over-regularization on Clean Data: While the model dominates in noisy environments, the heavy regularization means its performance gains on perfectly clean datasets are marginal compared to standard models.
  • Future Architectures: The authors suggest exploring how the RSTIB principle could be integrated with more specialized spatial-temporal modules beyond pure MLPs to capture even richer dynamics.

Summary updated

Abstract

Spatial-temporal forecasting (STF) plays a pivotal role in urban planning and computing. Spatial-Temporal Graph Neural Networks (STGNNs) excel at modeling spatial-temporal dynamics, thus being robust against noise perturbations. However, they often suffer from relatively poor computational efficiency. Simplifying the architectures can improve efficiency but also weakens robustness with respect to noise interference. In this study, we investigate the problem: *can simple neural networks such as Multi-Layer Perceptrons (MLPs) achieve robust spatial-temporal forecasting while remaining efficient?* To this end, we first reveal the *dual noise effect* in spatial-temporal data and propose a theoretically grounded principle termed *Robust Spatial-Temporal Information Bottleneck* (RSTIB), which holds strong potential for improving model robustness. We then design an implementation named *RSTIB-MLP*, together with a new training regime incorporating a knowledge distillation module, to enhance the robustness of MLPs for STF while maintaining their efficiency. Comprehensive experiments demonstrate that *RSTIB-MLP* achieves an excellent trade-off between robustness and efficiency, outperforming state-of-the-art STGNNs and MLP-based models. Our code is publicly available at: [https://github.com/mchen644/RSTIB](https://github.com/mchen644/RSTIB).
BibTeX
@inproceedings{chen2025information,
  author = {Min Chen and Guansong Pang and Wenjun Wang and Cheng Yan},
  title = {{Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=pqSLcEbk5X},
}
Figure