PDAGENT-BENCH: Characterizing, Grounding, and Architecting LLM/VLM Agents for VLSI Physical Design

Qiufeng Li, Rongqian Chen, Quan Cheng, Chengxuan Wang, Sizhe Tang, Chia-Tung Ho, David Z. Pan, Tian Lan, Weidong CaoarXiv 2026 · arXiv

TL;DR

This paper introduces PDAGENT-BENCH, the first comprehensive, multi-dimensional benchmark designed to evaluate LLM and Vision-Language Model (VLM) agents across the entire VLSI physical design stack. Alongside a suite of 353 curated tasks, the authors present PDAGENT, a closed-loop, multi-agent framework that uses expert "skill packs" to successfully drive end-to-end physical design flows. The evaluation reveals that while frontier models excel at conceptual reasoning, they struggle with exact tool syntax—though when augmented with human-engineered skills, VLMs can actually outperform state-of-the-art algorithmic placers.

What Problem Does This Paper Address?

While Large Language Models (LLMs) have successfully automated front-end chip design tasks like writing RTL (hardware description code), their application to back-end physical design remains largely underexplored. Physical design—the process of turning a logical circuit netlist into a physical, manufacturable layout geometry—is notoriously difficult. It involves multi-stage optimization under strict geometric constraints, complex tool orchestration (Electronic Design Automation or EDA tools), and long-latency feedback loops.

Prior benchmarks have been fragmented, focusing only on isolated subproblems like macro placement or basic script generation. Consequently, there has been no standardized way to evaluate how well an AI agent can handle the holistic, iterative, and feedback-driven nature of real-world physical design.

Methods

The authors tackle this gap with a two-pronged approach:

  1. PDAGENT-BENCH: A benchmark dataset containing 353 expert-validated problems spanning five core capability dimensions: foundational knowledge, report comprehension, root-cause analysis, script generation, and full-flow orchestration.
  2. PDAGENT Framework: A unified, multi-agent workflow architecture that mimics a human design team. It breaks down the workflow into specialized agent roles:
  3. Planner Agent: Decomposes user intent into a Directed Acyclic Graph (DAG) of subtasks.
  4. Worker Agent: Interacts directly with EDA tools via a persistent interactive shell.
  5. Analyzer Agent: Extracts quantitative metrics (like timing and power) from verbose tool logs.
  6. Debugger Agent: Diagnoses failures using root-cause analysis.
  7. Optimizer Agent: Tunes parameters based on metric trajectories.
[User Instruction] ➔ [Planner Agent] ➔ [Worker Agent] ⇄ [EDA Tools]
                             ▲              │
                             │              ▼
                      [Optimizer Agent] ◄ [Analyzer Agent]
                             ▲
                             │
                      [Debugger Agent]

A key novelty is the integration of Skill Packs—curated libraries encoding the procedural knowledge, reference scripts, and debugging strategies of senior engineers. Furthermore, to address the massive privacy concerns of using proprietary commercial PDKs (Process Design Kits) and EDA tools with cloud-based LLMs, the framework uses a local Python coordinator. This coordinator parses tool outputs locally, sending only sanitized, high-level metrics and error signatures to the LLM, ensuring the model never sees raw proprietary data.

Datasets

The paper introduces the PDAGENT-BENCH suite, which includes:

  • 90 Foundational Knowledge questions.
  • 11 Report Comprehension tasks.
  • 21 Root-Cause Analysis problems.
  • 11 Static Timing Analysis path-level reasoning tasks.
  • 210 Script Generation tasks split between commercial giants (90 for Cadence Innovus, 88 for Synopsys ICC2), Engineering Change Orders (10 ECO), and Formal Verification (22 FM).

To evaluate full-flow implementation, the authors test the agents on 10 diverse circuit designs ranging from small control blocks (like a 176-cell Elevator FSM) to massive industrial-scale designs (like the 1.5-million-cell NVIDIA Deep Learning Accelerator, NVDLA-large).

Key Insights and Ideas

  • Syntax is the Enemy, Not Concepts (Finding 1): An overwhelming 92% of script-generation failures were "near-misses." The LLMs correctly identified the right tool command but hallucinated or mismatched specific arguments and flags.
  • Perception Outpaces Action (Finding 3): Models are excellent at reading and interpreting EDA reports (~90% accuracy), but their performance drops significantly (~50%) when asked to diagnose the root cause or formulate an optimization strategy based on those reports.
  • The Power of Skill Augmentation (Finding 4): Standard Retrieval-Augmented Generation (RAG) baselines fail early in the design flow because general documentation is too coarse. Explicitly grounding agents with structured "skill packs" allows them to bypass fragile script generation and focus on high-level PPA (Power, Performance, Area) trade-offs.
  • VLMs Excel at Spatial Reasoning: When given visual layouts as inputs, vision-language models (VLMs) enhanced by human skills achieved superior Worst Negative Slack (WNS) and Total Negative Slack (TNS) in macro placement, outperforming specialized analytical and reinforcement learning-based placers.

Key Results

The authors evaluated 11 state-of-the-art models (including GPT-5 variants, Claude 4.x, and Qwen3-VL).

  • State-of-the-Art Baseline: GPT-5.5 emerged as the top performer, achieving 73.3% on root-cause analysis and 80.4% on foundational knowledge. However, even GPT-5.5 scored below 50% on raw script generation (45.5% for ICC2, 42.2% for Innovus), highlighting the difficulty of tool-centric execution.
  • End-to-End Success: Driven by a GPT-5-mini backbone, the full PDAGENT framework successfully completed the entire RTL-to-GDSII physical design flow on industrial designs (TinyRISCV, AES-256, Ethernet MAC) using a commercial TSMC 28nm process, successfully closing timing and resolving all design rule check (DRC) violations.

Limitations and Future Work

While a major milestone, the authors note several limitations:

  • Signoff Completeness: Layout-versus-Schematic (LVS) verification is currently excluded from the flow.
  • Automation Gaps: Certain tasks that heavily rely on GUI-based interactions or manual engineer inspection (like complex DRC fixing) are not yet fully automated.
  • Dataset Scale: The benchmark does not yet fully capture the extreme corner cases and heterogeneity of multi-billion-transistor, cutting-edge industrial chips.

Future Directions: The team plans to expand the benchmark to advanced nodes (16nm, 7nm) and mature nodes (65nm, 130nm), incorporate long-term memory and self-improvement modules so agents can learn from past failures, and integrate automated macro placement directly into the core framework loop.

Summary updated

Abstract

Large Language Models and vision-language models have shown remarkable success in the front-end design of Very Large-Scale Integrated Circuits, yet their capabilities for VLSI physical design remain significantly underexplored. The primary cause is the lack of standardized benchmarks for evaluating agentic physical design workflows that require high-dimensional, multi-stage optimization under strict design constraints, coordinated interaction with diverse Electronic Design Automation tools, and iterative refinement. This work introduces PDAGENT-BENCH, a comprehensive and multi-dimensional benchmark for evaluating LLM/VLM-based agents across the physical design stack. PDAGENT-BENCH integrates both task-level assessment and workflow-level execution. The benchmark suite contains 353 curated problems that combine conceptual questions with real-world industrial artifacts, with expert-validated references and executable solutions. In addition, the benchmark provides a unified, human-aligned agentic physical design workflow framework that enables closed-loop evaluation of holistic physical design in realistic EDA environments. Experiments on 11 state-of-the-art models reveal that while modern LLMs/VLMs perform competitively on conceptual tasks, they remain substantially limited in tool-centric execution (e.g., 42.2% on Innovus script generation) and long-horizon, multi-stage reasoning. Our studies further show that human-skill-enhanced agentic workflows significantly improve end-to-end physical design performance. PDAGENT-BENCH establishes a standardized, reproducible, and realistic evaluation framework for advancing LLM/VLM-driven holistic physical design automation. To ensure full reproducibility and broad accessibility, we will release PDAgent-Bench together with its agentic workflow framework, instantiated on open-source PDKs (e.g., Nangate45, ASAP7) and open EDA tools (e.g., OpenROAD).
BibTeX
@misc{li2026pdagent,
  author = {Qiufeng Li and Rongqian Chen and Quan Cheng and Chengxuan Wang and Sizhe Tang and Chia-Tung Ho and David Z. Pan and Tian Lan and Weidong Cao},
  title = {{PDAGENT-BENCH: Characterizing, Grounding, and Architecting LLM/VLM Agents for VLSI Physical Design}},
  year = {2026},
  eprint = {2606.17253},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.17253},
}

Loading related content...

Figure