Abstract
Spatial-temporal sequence forecasting (STSF) is critical for applications like traffic management and climate modeling but remains challenging due to complex spatial and temporal dependencies. While Neural Architecture Search (NAS) can automate model design, traditional methods often suffer from high computational costs or local optima. This paper introduces the ISTS (Instructive Neural Architecture Search) framework, which utilizes Large Language Models (LLMs) as expert instructors to guide the search process. By leveraging LLMs' internal knowledge through step-level, instance-level, and task-level interventions, ISTS achieves state-of-the-art performance with a 92% reduction in search time compared to gradient-based baselines.
Introduction
Predicting traffic flow or weather patterns requires balancing intricate relationships across space and time. Traditional NAS approaches face an "efficiency-performance" trade-off: heuristic methods are computationally expensive, while gradient-based methods like DARTS often converge to suboptimal, homogeneous structures. These methods are typically "data-blind," ignoring high-level structural logic that a human expert might use.
The ISTS framework addresses this by treating the LLM not as a simple code generator, but as a reasoning engine. LLMs possess vast internal "world knowledge" about deep learning architectures, which can be harnessed to navigate the search space more effectively than blind optimization algorithms.
The ISTS Framework
The ISTS framework enhances the LLM’s role across three distinct levels to optimize the architecture search process.
1. Step-level: Prompt Engineering
Instead of a single-shot generation, ISTS employs Chain-of-Thought (CoT) and Tree-of-Thought (ToT) prompting. This forces the LLM to decompose the task into logical steps: background comprehension, experience observation, and candidate generation, mimicking human expert reasoning.
2. Instance-level: Memory and Evaluation
To mitigate hallucinations and improve numerical reasoning, ISTS utilizes a Sorted Memory Bank. This component stores previous architecture attempts and their performance scores. By feeding this history back into prompts, the LLM learns to identify patterns in successful architectures without needing to perform precise numerical calculations. A "one-step tuning" framework provides rapid, preliminary evaluations to keep the search efficient.
3. Task-level: Two-stage Search
The search process is divided into two phases:
- Exploration Stage: Prioritizes diversity and novel operator combinations to avoid local optima.
- Optimization Stage: Refines the most promising candidates to maximize performance.
The search space is defined as a Directed Acyclic Graph (DAG) using three specialized cell types: Spatial-Temporal-Parallel (STP), Spatial-then-Temporal (STT), and Temporal-then-Spatial (TTS).
Experimental Evaluation
The framework was tested on five real-world traffic forecasting benchmarks: PEMS-03, PEMS-04, PEMS-08 (California traffic flow), and PEMS-BAY, METR-LA (traffic speed). These datasets involve hundreds of sensors and extensive temporal sequences.
Key Results
- Performance: ISTS achieved the best performance in 9 out of 15 fixed metrics across the five datasets, outperforming both manual models (e.g., GraphWaveNet) and established NAS baselines (e.g., DARTS and PC-DARTS).
- Efficiency: On the PEMS-08 dataset, the search process for ISTS-COT took only 68 seconds total, compared to 887 seconds per epoch for DARTS, representing a reduction in processing time of over 90%.
- Robustness: The framework proved robust across different LLMs, including LLaMA-3 (8B), Qwen-2 (7B), and even the smaller Phi-3-mini (3.8B), showing that the instructional framework is more critical than the raw size of the model.
Discussion and Conclusion
ISTS demonstrates that LLMs can serve as effective "expert instructors" in specialized domains like STSF. By providing a structured reasoning framework and a memory of past performance, the system overcomes the LLM's inherent weaknesses in numerical precision. Future work will explore expanding the operator search space beyond standard STSF cells and integrating the LLM more deeply into weight-sharing mechanisms to further bridge the gap between instructing and executing the search.
Summary updated