UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines

Chen Tang, Xinzhu Ma, Encheng Su, Xiufeng Song, Xiaohong Liu, Wei-Hong Li, Lei Bai, Wanli Ouyang, Xiangyu Yue

TL;DR

UniSTD is a unified Transformer-based framework that breaks the "one-model-per-task" paradigm in spatiotemporal learning. By leveraging 2D vision pretraining and a novel rank-adaptive Mixture-of-Experts (MoE) mechanism, it successfully handles 10 diverse tasks—ranging from weather forecasting to traffic control—within a single model, often outperforming specialized state-of-the-art architectures.

The Challenge: Siloed Spatiotemporal Learning

Spatiotemporal learning, which involves predicting future states based on historical data, is vital for applications like autonomous driving and climate modeling. However, the field has traditionally relied on specialized architectures. A model built for weather forecasting usually performs poorly if applied to human motion prediction.

Conceptual overview comparing the traditional siloed approach of specialized models with the unified UniSTD framework.

This "siloed" approach has three major drawbacks:

  1. Limited Generalizability: Models do not transfer well across domains.
  2. High Costs: Training new models from scratch for every task is computationally expensive.
  3. Missed Opportunities: Specialized models cannot benefit from "cross-task learning," where knowledge from one domain (e.g., physics in weather) might help another (e.g., fluid-like traffic flow).

Methodology: The UniSTD Framework

UniSTD adopts a two-stage "pretraining-then-adaptation" paradigm, common in Large Language Models but novel in this domain. Its architecture is defined by three core innovations:

  • Standard Transformer Backbone: Instead of custom blocks, UniSTD uses a standard Vision Transformer (ViT) initialized with weights from large-scale 2D vision/text datasets (like OpenCLIP). This proves that 2D spatial knowledge is a powerful foundation for 4D (3D space + time) prediction.
  • Rank-Adaptive Mixture-of-Experts (RA-MoE): Different tasks require different levels of model complexity. UniSTD uses Low-Rank Adaptation (LoRA) but makes the "rank" dynamic. Using a continuous relaxation technique, the model learns the optimal rank for each task during training.
  • Lightweight Temporal Module: To help the 2D-pretrained Transformer understand time, a small, zero-initialized temporal attention module is added to capture frame-to-frame dynamics without overwriting spatial features.

Datasets and Evaluation

The framework was evaluated on 10 tasks across 4 disciplines:

  • Traffic Control: TaxiBJ (crowd flow) and Traffic4Cast (city-wide traffic).
  • Trajectory & Robot Planning: Moving MNIST, BAIR (robot pushing), Human3.6M (human motion), and KTH Action.
  • Driving Scene Prediction: Cityscapes and KITTI.
  • Weather Forecasting: SEVIR (radar/satellite) and ENSO (El Niño prediction).

Key Results and Insights

  • Superior Multi-Task Performance: In a joint-training scenario, UniSTD outperformed existing methods like SimVP and TAU by massive margins—up to 18.8 PSNR in some cases.
  • Efficiency and Scaling: UniSTD uses roughly 50% fewer trainable parameters than specialized baselines. While traditional models often see performance drops when moving from 1 task to 3, UniSTD maintained or improved performance as more tasks were added, demonstrating true cross-task synergy.
  • 2D Pretraining as a "Free Lunch": The paper demonstrates that models pretrained on static images provide a much better starting point for video-like prediction than training from scratch, even for non-natural images like weather radar.

Limitations and Future Work

While UniSTD handles 10 tasks well, a significant domain gap remains between synthetic data and complex real-world data. Additionally, like many Transformer-based predictors, the model may face challenges with very long-term horizons where errors can accumulate. Future research may incorporate more modalities, such as text descriptions, to further enhance the model's physical understanding.

Summary updated

BibTeX
@inproceedings{tang2025unistd,
  author = {Chen Tang and Xinzhu Ma and Encheng Su and Xiufeng Song and Xiaohong Liu and Wei-Hong Li and Lei Bai and Wanli Ouyang and Xiangyu Yue},
  title = {{UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02720},
}
Figure