Manticore: Hardware-Accelerated RTL Simulation with Static Bulk-Synchronous Parallelism

Mahyar Emami, Sahand Kashani, Keisuke Kamahori, Mohammad Sepehr Pourghannad, Ritik Raj, James R. Larus

Introduction

Hardware designers are currently caught in a "simulation dilemma." On one end of the spectrum, software simulators like Verilator offer rapid compilation and high visibility but suffer from slow execution speeds. On the other end, FPGA prototypes provide near-real-time execution but require hours or even days for synthesis and place-and-route, severely hindering iterative debugging.

The fundamental bottleneck in software simulation is the mismatch between modern CPU architectures and the nature of Register-Transfer Level (RTL) designs. RTL consists of thousands of small, independent logic gates that must synchronize every clock cycle. On general-purpose multicore CPUs, the overhead of managing communication and synchronization for these "tiny tasks" is so significant that simulators often perform best when restricted to a single core, failing to leverage the potential of parallel hardware.

The Manticore Architecture

Manticore is a specialized hardware accelerator designed to bridge this gap. It utilizes a specialized architecture and a custom compiler stack to exploit fine-grained parallelism without the traditional overhead of software-based synchronization. The system is built upon four primary pillars:

1. Static Bulk-Synchronous Parallel (BSP) Execution

Unlike standard parallel models that rely on runtime barriers or locks, Manticore employs a static BSP model. The compiler generates a precise, cycle-by-cycle schedule for every core in the system. Because RTL code typically lacks the divergent execution paths found in general-purpose software, the compiler can predict exactly when data will be ready and when it must be transmitted. This approach entirely eliminates runtime synchronization overhead.

2. Deterministic Hardware Design

The Manticore "Machine" consists of a grid of simple, 16-bit cores interconnected by a 2D torus Network-on-Chip (NoC). To ensure the compiler's schedule remains accurate, the cores are designed for total predictability; they lack complex features like branch predictors or out-of-order execution units that introduce non-deterministic timing.

3. Custom Function Synthesis

To handle the bitwise logic dominant in RTL, Manticore’s compiler identifies chains of logic operations and collapses them into custom instructions. These are executed by a 4-input lookup table (LUT) unit within each core, significantly reducing the total instruction count compared to standard RISC instructions.

4. Global Stall Mechanism

To accommodate non-deterministic events, such as off-chip DRAM access, Manticore utilizes a "global stall" implemented via FPGA clock gating. When a memory access is required, the system pauses the clock for the entire compute grid. This keeps all cores in perfect lockstep, preserving the static schedule without requiring complex handshake logic.

Evaluation and Methodology

The researchers evaluated Manticore using a prototype of 225 cores implemented on a single Xilinx Alveo U200 FPGA running at 475 MHz. The system was tested against nine diverse RTL benchmarks, including a Bitcoin miner (bc), a 16x16 matrix multiplier (mm), a RISC-V processor cluster (rv32r), and a JPEG decoder (jpeg).

Performance was compared against Verilator v5.006 running on high-end hardware, including Intel i7-9700K, Xeon 8272CL, and AMD EPYC 7V73X processors.

Key Results and Insights

Performance and Scaling

Manticore outperformed Verilator in 8 out of the 9 benchmarks. It achieved up to an 18x speedup over serial Verilator and up to a 6.8x speedup over multithreaded Verilator. Notably, while software simulators often see diminishing returns after 4 to 8 threads, Manticore demonstrated continued performance scaling as it utilized more than 200 cores.

Efficiency of Coordination

The primary insight of the Manticore project is that the bottleneck in RTL simulation is not computation, but the cost of coordination. By moving coordination to compile-time through static scheduling and predication (executing all conditional paths and selecting the result), Manticore makes tasks only a few instructions long viable for parallel execution.

Cost-Effectiveness

For large-scale simulations (e.g., 10 billion cycles), Manticore is not only faster but potentially more cost-effective in cloud environments. The ability to pack hundreds of specialized 16-bit cores onto an FPGA provides a higher throughput-per-dollar ratio than renting high-core-count general-purpose CPUs for the same task.

Limitations and Future Work

Despite its advantages, the current Manticore prototype has several limitations:

  • Clock Domains: It currently only supports designs with a single global clock.
  • Language Support: The compiler does not yet support the full SystemVerilog standard.
  • Amdahl’s Law: Benchmarks with heavy sequential dependencies, such as Huffman decoding in the JPEG test, see lower speedups because they cannot be effectively parallelized.
  • Compile Times: While faster than full FPGA synthesis, Manticore's compiler is currently slower than Verilator's (minutes versus seconds).

Future research will focus on supporting multiple clock domains, implementing out-of-band waveform collection for debugging, and exploring heterogeneous core grids to better handle memory-intensive tasks.

Summary updated

BibTeX
@inproceedings{emami2023manticore,
  author = {Mahyar Emami and Sahand Kashani and Keisuke Kamahori and Mohammad Sepehr Pourghannad and Ritik Raj and James R. Larus},
  title = {{Manticore: Hardware-Accelerated RTL Simulation with Static Bulk-Synchronous Parallelism}},
  booktitle = {ASPLOS},
  year = {2023},
  doi = {10.1145/3623278.3624750},
}
Figure