ISAAC: Intelligent, Scalable, Agile, and Accelerated CPU Verification via LLM-aided FPGA Parallelism

Nan GuanarXiv 2025 · DOI · arXiv

TL;DR

CPU verification is a major bottleneck in chip development, often taking 70% of project time. This paper introduces ISAAC, a novel framework that leverages Large Language Models (LLMs) for intelligent stimulus generation and FPGA parallelism for accelerated verification, achieving significant speedups and uncovering previously unknown bugs.

What Problem Does This Paper Address?

Functional verification of CPUs is notoriously time-consuming and labor-intensive. Current industry practices, like differential testing, face several challenges:

  • Front-end Stimulus Generation: Existing methods often lack awareness of the CPU's micro-architecture, leading to redundant tests, slow coverage closure, and missed corner cases.
  • Back-end Simulation Infrastructure: Even with FPGA acceleration, long-running tests create bottlenecks, and limited visibility hinders debugging. The tight coupling between the Instruction Set Simulator (ISS) and the Design Under Test (DUT) forces the fast ISS to wait for the slower DUT, creating significant overhead.
  • Coverage vs. Correctness: High code coverage doesn't guarantee functional correctness, as subtle bugs can persist and only be found through extensive, often manual, testing.

Methods

ISAAC tackles these challenges with a two-pronged approach:

1. Intelligent Stimulus Generation (Front-end): * LLM-Aided Multi-Agent System: ISAAC employs LLM agents infused with micro-architectural knowledge and historical bug patterns. This allows for the generation of highly targeted tests that proactively address potential issues and achieve coverage goals faster. * Two-Stage Workflow: * Coverage-Augmented Testing: This stage uses LLMs to analyze coverage reports and generate refined stimulus to fill identified gaps. It incorporates a "Report Distillation" mechanism to make coverage reports more digestible for LLMs. * Post-Coverage Stress Testing: Once coverage plateaus, this stage leverages a historical bug database and LLMs to generate tests specifically designed to trigger known and complex corner-case bugs. * Signal Waiving: ISAAC intelligently identifies and waives irrelevant signals from coverage metrics to focus on architecturally meaningful behaviors, preventing misleading coverage reports.

2. Accelerated FPGA Parallelism (Back-end): * Decoupled ISS-DUT Execution: ISAAC breaks the traditional lockstep execution model. The ISS runs ahead, charting program flow, and captures architectural states at adaptive checkpoints. * Lightweight Forward-Snapshot Mechanism: Instead of full state dumps, ISAAC captures only essential architectural registers and memory access information, significantly reducing snapshot size and transfer overhead. * Parallel Replay: These lightweight snapshots enable multiple DUTs on an FPGA cluster to replay independent segments of a workload concurrently, unlocking both intra-workload (within a single program) and inter-workload (across different programs) parallelism. * Micro-architectural Add-on: A non-invasive hardware add-on (Replay Control Unit, Memory Access Context, Segment Result Checker) is integrated into the DUT to manage replay and perform differential checking.

Datasets

  • Existing Benchmarks: The paper mentions using standard test suites like riscv-dv [15, 43], architectural benchmarks like Embench [40], and fuzzing techniques.
  • Historical Bug Database: ISAAC constructs and utilizes a taxonomy-based database of common CPU errors, drawing from prior research, public errata, and industry reports. This database is crucial for grounding LLM prompts in validated failure patterns.
  • No new datasets were introduced for training LLMs. The LLMs (specifically GPT-4) are used in an inference capacity, guided by the existing data and the framework's analysis.

Key Insights and Ideas

  • LLMs as "Expert Intuition": The core insight is that LLMs, when properly guided with architectural knowledge and historical bug data, can mimic expert engineers' intuition to generate targeted, high-value tests, overcoming the limitations of purely random or coverage-guided fuzzing.
  • Decoupling for Speed: Breaking the tight lockstep between ISS and DUT is crucial for achieving significant speedups. Allowing the ISS to run ahead and using lightweight snapshots unlocks parallelism that was previously hindered by synchronization overhead.
  • Coverage is Necessary, Not Sufficient: The paper emphasizes that while coverage is important, it doesn't guarantee bug detection. ISAAC's dual-stage approach addresses this by first maximizing coverage and then specifically targeting known and complex bugs.
  • Adaptive Strategies are Key: Both stimulus generation (adapting to coverage gaps) and back-end execution (adaptive segment granularity and MAC window size) benefit from adaptive strategies to optimize performance and resource utilization.

Key Results

  • Coverage: ISAAC achieved 91% code coverage and 100% Known Bug Trigger Coverage (KBTC) in less than half the runtime of the best baseline. It also demonstrated higher coverage per input compared to fuzzing methods like PathFuzz.
  • Bug Triggering: ISAAC achieved 45.1 triggers per 100K instructions, a 1.7x improvement over UVM² and 2.3x over DiFuzzRTL, indicating higher effectiveness in exposing known bugs.
  • Performance: ISAAC achieved up to a 17,536× speed-up over software RTL simulation (specifically over MinJie) and a 13.0× speed-up over Hassert (an advanced prior FPGA-based framework).
  • Bug Discovery: ISAAC uncovered several previously unknown bugs, two of which are detailed in the paper (Memory Access Deadlock and Read-Write Conflict in Dual-Port BTB SRAM).
  • Hardware Cost: The overhead for in-chip checking circuits was modest, adding only 2.4% LUTs, 0.9% LUTRAMs, and 3.7% Flip-Flops relative to a pure Rocket core.

Limitations and Future Work

  • Limitations:
    • Communication Overhead: While significantly reduced, data transmission for memory access still represents a bottleneck, contributing 86.5% of overhead in certain configurations.
  • Scalability Plateau: Performance gains from increasing the number of DUTs plateau around 16-20 instances.
  • Segment Granularity Sensitivity: Performance can degrade with very coarse or very fine segment granularities, depending on workload type.
  • Future Work:
    • Further optimization of communication overhead.
  • Exploring more advanced LLM prompting and fine-tuning strategies.
  • Investigating more sophisticated adaptive management techniques for segment generation.
  • Extending ISAAC to other complex IP verification scenarios beyond CPUs.

Summary updated

BibTeX
@misc{guan2025isaac,
  author = {Nan Guan},
  title = {{ISAAC: Intelligent, Scalable, Agile, and Accelerated CPU Verification via LLM-aided FPGA Parallelism}},
  year = {2025},
  eprint = {2510.10225},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10225},
}
Figure