Instructing the Architecture Search for Spatial-temporal Sequence Forecasting with LLM

Xin Xue, Haoyi Zhou, Tianyu Chen, Shuai Zhang, Yizhou Long, Jianxin LiarXiv 2025 · DOI · arXiv

Abstract

Spatial-temporal sequence forecasting (STSF) is critical for applications like traffic management and climate modeling but remains challenging due to complex spatial and temporal dependencies. While Neural Architecture Search (NAS) can automate model design, traditional methods often suffer from high computational costs or local optima. This paper introduces the ISTS (Instructive Neural Architecture Search) framework, which utilizes Large Language Models (LLMs) as expert instructors to guide the search process. By leveraging LLMs' internal knowledge through step-level, instance-level, and task-level interventions, ISTS achieves state-of-the-art performance with a 92% reduction in search time compared to gradient-based baselines.

Introduction

Predicting traffic flow or weather patterns requires balancing intricate relationships across space and time. Traditional NAS approaches face an "efficiency-performance" trade-off: heuristic methods are computationally expensive, while gradient-based methods like DARTS often converge to suboptimal, homogeneous structures. These methods are typically "data-blind," ignoring high-level structural logic that a human expert might use.

The ISTS framework addresses this by treating the LLM not as a simple code generator, but as a reasoning engine. LLMs possess vast internal "world knowledge" about deep learning architectures, which can be harnessed to navigate the search space more effectively than blind optimization algorithms.

The ISTS Framework

The ISTS framework enhances the LLM’s role across three distinct levels to optimize the architecture search process.

The ISTS framework operates at three levels: step-level instruction, instance-level memory, and task-level search stages.

1. Step-level: Prompt Engineering

Instead of a single-shot generation, ISTS employs Chain-of-Thought (CoT) and Tree-of-Thought (ToT) prompting. This forces the LLM to decompose the task into logical steps: background comprehension, experience observation, and candidate generation, mimicking human expert reasoning.

2. Instance-level: Memory and Evaluation

To mitigate hallucinations and improve numerical reasoning, ISTS utilizes a Sorted Memory Bank. This component stores previous architecture attempts and their performance scores. By feeding this history back into prompts, the LLM learns to identify patterns in successful architectures without needing to perform precise numerical calculations. A "one-step tuning" framework provides rapid, preliminary evaluations to keep the search efficient.

The search process is divided into two phases:

  • Exploration Stage: Prioritizes diversity and novel operator combinations to avoid local optima.
  • Optimization Stage: Refines the most promising candidates to maximize performance.

The search space is defined as a Directed Acyclic Graph (DAG) using three specialized cell types: Spatial-Temporal-Parallel (STP), Spatial-then-Temporal (STT), and Temporal-then-Spatial (TTS).

Example of a Directed Acyclic Graph (DAG) architecture generated within the ISTS search space.

Experimental Evaluation

The framework was tested on five real-world traffic forecasting benchmarks: PEMS-03, PEMS-04, PEMS-08 (California traffic flow), and PEMS-BAY, METR-LA (traffic speed). These datasets involve hundreds of sensors and extensive temporal sequences.

Key Results

  • Performance: ISTS achieved the best performance in 9 out of 15 fixed metrics across the five datasets, outperforming both manual models (e.g., GraphWaveNet) and established NAS baselines (e.g., DARTS and PC-DARTS).
  • Efficiency: On the PEMS-08 dataset, the search process for ISTS-COT took only 68 seconds total, compared to 887 seconds per epoch for DARTS, representing a reduction in processing time of over 90%.
  • Robustness: The framework proved robust across different LLMs, including LLaMA-3 (8B), Qwen-2 (7B), and even the smaller Phi-3-mini (3.8B), showing that the instructional framework is more critical than the raw size of the model.

Discussion and Conclusion

ISTS demonstrates that LLMs can serve as effective "expert instructors" in specialized domains like STSF. By providing a structured reasoning framework and a memory of past performance, the system overcomes the LLM's inherent weaknesses in numerical precision. Future work will explore expanding the operator search space beyond standard STSF cells and integrating the LLM more deeply into weight-sharing mechanisms to further bridge the gap between instructing and executing the search.

Summary updated

Abstract

Spatial-temporal sequence forecasting (STSF) is a long-standing research problem with widespread real-world applications. Neural architecture search (NAS), which automates the neural network design, has been shown effective in tackling the STSF problem. However, the existing NAS methods for STSF focus on generating architectures in a time-consuming data-driven fashion, which heavily limits their ability to use background knowledge and explore the complicated search trajectory. Large language models (LLMs) have shown remarkable ability in decision-making with comprehensive internal world knowledge, but how it could benefit NAS for STSF remains unexplored. In this paper, we propose a novel NAS method for STSF based on LLM. Instead of directly generate architectures with LLM, We inspire the LLM's capability with a multi-level enhancement mechanism. Specifically, on the step-level, we decompose the generation task into decision steps with powerful prompt engineering and inspire LLM to serve as instructor for architecture search based on its internal knowledge. On the instance-level, we utilize a one-step tuning framework to quickly evaluate the architecture instance and a memory bank to cumulate knowledge to improve LLM's search ability. On the task-level, we propose a two-stage architecture search, balancing the exploration stage and optimization stage, to reduce the possibility of being trapped in local optima. Extensive experimental results demonstrate that our method can achieve competitive effectiveness with superior efficiency against existing NAS methods for STSF.
BibTeX
@misc{xue2025instructing,
  author = {Xin Xue and Haoyi Zhou and Tianyu Chen and Shuai Zhang and Yizhou Long and Jianxin Li},
  title = {{Instructing the Architecture Search for Spatial-temporal Sequence Forecasting with LLM}},
  year = {2025},
  eprint = {2503.17994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.17994},
}
Figure