SMARTS

Roland E. Wunderlich, Thomas F. Wenisch, Babak Falsafi, James C. Hoe

TL;DR

This paper introduces SMARTS, a novel framework for accelerating microarchitecture simulation by using statistically sound sampling. SMARTS allows for accurate performance estimations of full benchmarks with significantly reduced simulation time, achieving speedups of up to 60x over detailed simulation while maintaining very low error rates.

What Problem Does This Paper Address?

Designing modern processors involves extensive simulation to evaluate new architectural ideas. However, current cycle-accurate simulators are incredibly slow, often taking days or weeks to simulate just one minute of real-world execution. To cope, researchers often resort to simulating only small portions of benchmark programs. This approach is problematic because these truncated simulations can be inaccurate and misleading, failing to capture the full spectrum of a program's behavior. The core challenge is to achieve both speed and accuracy in microarchitecture simulation, bridging the gap between the slow, accurate detailed simulations and the fast, but potentially inaccurate, truncated simulations.

Methods

The SMARTS framework tackles this problem by employing a statistically rigorous approach to "simulation sampling." Instead of simulating an entire benchmark, SMARTS simulates only carefully selected "sampling units" in detail, while "fast-forwarding" through the rest of the execution using a much faster functional simulator.

The key innovations are:

  • Systematic Sampling: SMARTS uses systematic sampling, where sampling units are chosen at regular intervals, making it easier to implement in simulators.
  • Statistical Confidence: The framework prescribes a statistically sound procedure to determine the minimal number of instructions to simulate to achieve a desired confidence interval (e.g., ±3% error with 99.7% confidence). This is achieved by analyzing the "coefficient of variation" (a measure of how much a metric varies within the benchmark).
  • Microarchitectural State Warming: A critical challenge in sampling is ensuring that the microarchitectural state (like branch predictors and cache contents) is accurate when a sampling unit begins. SMARTS addresses this with "detailed warming" (simulating a short period in detail before measurement) and, more importantly, "functional warming" (using a fast functional simulator to maintain key architectural state). Functional warming is crucial for achieving high speedups.

Datasets

The paper evaluates SMARTS using the SPEC CPU2000 (SPEC2K) integer and floating-point benchmarks. They analyze 41 out of 45 possible benchmark/input combinations. The study uses two microarchitecture configurations: an 8-way superscalar processor (baseline) and a 16-way superscalar processor (aggressive future design).

Key Insights and Ideas

  • Statistical Rigor is Key: The paper emphasizes that simply picking random chunks of code is insufficient. A statistically grounded method is needed to guarantee accuracy and confidence.
  • Variability is Predictable (to an extent): The "coefficient of variation" of metrics like CPI is a powerful predictor of how much sampling is needed. Benchmarks with low variability require less sampling.
  • Functional Warming is a Game-Changer: While detailed warming is necessary, functional warming is what truly enables high speedups by allowing for very small detailed warming periods, effectively decoupling the sampling rate from the detailed simulator's speed.
  • Small Sampling Units are Better: Contrary to intuition, using many small sampling units (e.g., 1000 instructions) is more effective than fewer large ones for achieving accuracy with minimal detailed simulation.
  • The "Optimal" U is Often 1000: While an "optimal" sampling unit size (U) can be calculated, the paper suggests that U=1000 is a good general-purpose choice that balances accuracy and simulation effort across most benchmarks.

Key Results

  • High Accuracy: SMARTS achieved an actual average error of only 0.64% on CPI and 0.59% on EPI for the tested benchmarks.
  • Significant Speedups: SMARTS provided average speedups of 35x for the 8-way configuration and 60x for the 16-way configuration compared to full detailed simulation. This translates to simulation speeds of over 9 MIPS on a 2 GHz Pentium 4.
  • Minimal Simulation: For a stringent ±3% error with 99.7% confidence, SMARTS required measuring fewer than 50 million instructions per benchmark, which is a tiny fraction of the total benchmark streams (often billions of instructions).
  • Bias Management: The paper quantifies the impact of microarchitectural state warming bias, showing that functional warming significantly reduces it, keeping residual bias below 2% for most benchmarks.

Limitations and Future Work

  • Bias Uncertainty: While functional warming helps, residual bias from imperfect microarchitectural state warming can still introduce uncertainty. The paper empirically bounds this bias but acknowledges it's not analytically determined for all cases.
  • SimPoint Comparison: The paper compares SMARTS to SimPoint, another sampling technique. While SimPoint achieved slightly faster runtimes in their comparison, SMARTS demonstrated significantly lower CPI error and offered quantifiable confidence intervals, which SimPoint lacked. This suggests a trade-off between speed and guaranteed accuracy.
  • Future Simulator Design: The authors suggest that future simulator designs should focus less on optimizing detailed simulators themselves and more on improving fast-forwarding and functional warming techniques, as these are the true bottlenecks for sampling-based simulation.

Summary updated

Abstract

Current software-based microarchitecture simulators are many orders of magnitude slower than the hardware they simulate. Hence, most microarchitecture design studies draw their conclusions from drastically truncated benchmark simulations that are often inaccurate and misleading. This paper presents the Sampling Microarchitecture Simulation (SMARTS) framework as an approach to enable fast and accurate performance measurements of full-length benchmarks. SMARTS accelerates simulation by selectively measuring in detail only an appropriate benchmark subset. SMARTS prescribes a statistically sound procedure for configuring a systematic sampling simulation run to achieve a desired quantifiable confidence in estimates.Analysis of 41 of the 45 possible SPEC2K benchmark/input combinations show CPI and energy per instruction (EPI) can be estimated to within ±3% with 99.7% confidence by measuring fewer than 50 million instructions per benchmark. In practice, inaccuracy in microarchitectural state initialization introduces an additional uncertainty which we empirically bound to ∼2% for the tested benchmarks. Our implementation of SMARTS achieves an actual average error of only 0.64% on CPI and 0.59% on EPI for the tested benchmarks, running with average speedups of 35 and 60 over detailed simulation of 8-way and 16-way out-of-order processors, respectively.
BibTeX
@inproceedings{wunderlich2003smarts,
  author = {Roland E. Wunderlich and Thomas F. Wenisch and Babak Falsafi and James C. Hoe},
  title = {{SMARTS}},
  booktitle = {ACM SIGARCH Computer Architecture News},
  year = {2003},
  doi = {10.1145/871656.859629},
}
Figure