TL;DR
This paper introduces a "world-model-inspired" framework for tensor program optimization that treats code scheduling as a sequence of action-conditioned latent state transitions rather than a series of static code snapshots. By rolling out scheduling decisions entirely within a continuous latent space, the framework captures complex transformation dependencies and filters out syntactic noise, achieving search quality comparable to state-of-the-art auto-schedulers like Ansor while requiring 10× fewer expensive hardware measurements.
What Problem Does This Paper Address?
To run deep learning models efficiently across diverse hardware (CPUs, GPUs, accelerators), tensor compilers must find the optimal "schedule"—the exact sequence of loop transformations, tiling, memory placements, and parallelization choices for a given workload. Because this search space is combinatorially massive and direct on-device measurement is slow, compilers rely on learned cost models to evaluate and rank candidate programs.
However, existing cost models suffer from a fundamental flaw: they treat candidate programs as static code snapshots, completely ignoring the step-by-step transformation history that generated them.
This static view causes two major issues:
- Syntax Sensitivity: Minor variations in workload sizes (e.g., a matrix dimension of 1023 vs. 1024) can force the compiler to generate extra boundary-handling loops ("loop peeling"). Even though the core computation and hardware behaviors are virtually identical, the resulting code texts look completely different, misleading static code encoders.
- Referential Ambiguity: Scheduling actions are context-dependent. For example, an action might fuse two loops to create a new loop variable $k$, which a subsequent action then tiles. A static or action-only model cannot easily resolve what entity $k$ actually represents without tracking the intermediate states.
Methods
The authors address this by framing tensor program optimization as a world model problem, where the compiler and hardware act as the environment, TensorIR representations act as states, and scheduling choices act as actions.
Instead of mutating the actual Abstract Syntax Tree (AST) or generating text at every step, the framework simulates the entire scheduling trajectory in a continuous latent space using three core components:
- TensorIR State Encoder: A trainable CodeBERT-based model adapted for long inputs via a sliding-window chunking mechanism. It uses contrastive learning to map initial unoptimized program states ($s_0$) into a dense latent space ($z_0$).
- Action-Conditioned Multi-Step Transition Model: Inspired by knowledge graph representation learning (specifically TransH), this lightweight geometric model recurrently applies scheduling actions directly to the latent state ($z_{t+1} = F(z_t, a_t)$). This allows the model to efficiently predict the final terminal-state representation ($\hat{z}_T$) while tracking the provenance of every decision.
- Ranking-Based Cost Model: An XGBoost model trained with a LambdaRank objective. Instead of predicting absolute execution time, it takes the predicted terminal latent state, structured action features, and hardware features to rank candidate schedules within the same workload.
Datasets
The framework was trained and evaluated using:
- TenSet Tuning Logs: A large-scale program performance dataset for tensor compilers. The authors extracted 10,000 samples for contrastive state-representation learning, nearly 400,000 state-action pairs for transition modeling, and up to 500,000 measured samples for the ranking model.
- Evaluation Benchmarks: 103 subgraphs extracted from 7 representative deep learning models covering CNNs, Transformers, and 3D vision (including ResNet-50, MobileNetV2, BERT, GPT-2, and OPT-1.3B). These were further grouped into 22 representative subgraph types for fine-grained analysis.
Key Insights and Ideas
- Latent Space Rollouts Bypass Compiler Overhead: Simulating code transformations mathematically in a latent space avoids the heavy computational overhead of materializing intermediate ASTs or text strings during active search.
- Suppression of Syntactic Noise: By learning transition dynamics rather than static text patterns, the model learns to group semantically equivalent programs together, making it robust against superficial syntax changes like loop peeling.
- Prioritizing Ranking over Regression: Predicting absolute execution times across different hardware and workloads is notoriously difficult. Shifting the objective to intra-workload ranking perfectly aligns the model with the auto-scheduler's actual task: prioritizing the best candidates for physical measurement.
Key Results
The framework was integrated into TVM AutoScheduler and tested on an Intel Xeon CPU and an NVIDIA RTX 4090 GPU:
- High Sample Efficiency: With a budget of just 1,000 trials, the method matched the search quality of Ansor running for 10,000 trials (within 2.2% geometric mean latency), yielding a 10× reduction in tuning measurements.
- Substantial Speedups: Under a strict 64-trial budget, it improved representative-subgraph latency over Ansor by 1.37× on GPU and 1.54× on CPU.
- End-to-End Acceleration: For full-model inference, the optimized programs outperformed standard PyTorch by 4.61× and cuDNN-enabled PyTorch-opt by 3.67× (geometric mean).
Limitations and Future Work
- Bounded by Candidate Generation: This framework is purely an evaluator. If the underlying auto-scheduler fails to propose high-quality candidate schedules in its search space, a better evaluator cannot magically recover them.
- Compounding Errors: Because the terminal state is reached via multi-step rollouts, prediction errors can accumulate over exceptionally long or complex scheduling trajectories.
- No Cross-Workload Calibration: Because the model uses a relative ranking objective within individual workloads, its scores cannot be used to compare absolute performance across entirely different models or distinct hardware targets.
Summary updated