TL;DR
This paper introduces PDAGENT-BENCH, the first comprehensive, multi-dimensional benchmark designed to evaluate LLM and Vision-Language Model (VLM) agents across the entire VLSI physical design stack. Alongside a suite of 353 curated tasks, the authors present PDAGENT, a closed-loop, multi-agent framework that uses expert "skill packs" to successfully drive end-to-end physical design flows. The evaluation reveals that while frontier models excel at conceptual reasoning, they struggle with exact tool syntax—though when augmented with human-engineered skills, VLMs can actually outperform state-of-the-art algorithmic placers.
What Problem Does This Paper Address?
While Large Language Models (LLMs) have successfully automated front-end chip design tasks like writing RTL (hardware description code), their application to back-end physical design remains largely underexplored. Physical design—the process of turning a logical circuit netlist into a physical, manufacturable layout geometry—is notoriously difficult. It involves multi-stage optimization under strict geometric constraints, complex tool orchestration (Electronic Design Automation or EDA tools), and long-latency feedback loops.
Prior benchmarks have been fragmented, focusing only on isolated subproblems like macro placement or basic script generation. Consequently, there has been no standardized way to evaluate how well an AI agent can handle the holistic, iterative, and feedback-driven nature of real-world physical design.
Methods
The authors tackle this gap with a two-pronged approach:
- PDAGENT-BENCH: A benchmark dataset containing 353 expert-validated problems spanning five core capability dimensions: foundational knowledge, report comprehension, root-cause analysis, script generation, and full-flow orchestration.
- PDAGENT Framework: A unified, multi-agent workflow architecture that mimics a human design team. It breaks down the workflow into specialized agent roles:
- Planner Agent: Decomposes user intent into a Directed Acyclic Graph (DAG) of subtasks.
- Worker Agent: Interacts directly with EDA tools via a persistent interactive shell.
- Analyzer Agent: Extracts quantitative metrics (like timing and power) from verbose tool logs.
- Debugger Agent: Diagnoses failures using root-cause analysis.
- Optimizer Agent: Tunes parameters based on metric trajectories.
[User Instruction] ➔ [Planner Agent] ➔ [Worker Agent] ⇄ [EDA Tools]
▲ │
│ ▼
[Optimizer Agent] ◄ [Analyzer Agent]
▲
│
[Debugger Agent]
A key novelty is the integration of Skill Packs—curated libraries encoding the procedural knowledge, reference scripts, and debugging strategies of senior engineers. Furthermore, to address the massive privacy concerns of using proprietary commercial PDKs (Process Design Kits) and EDA tools with cloud-based LLMs, the framework uses a local Python coordinator. This coordinator parses tool outputs locally, sending only sanitized, high-level metrics and error signatures to the LLM, ensuring the model never sees raw proprietary data.
Datasets
The paper introduces the PDAGENT-BENCH suite, which includes:
- 90 Foundational Knowledge questions.
- 11 Report Comprehension tasks.
- 21 Root-Cause Analysis problems.
- 11 Static Timing Analysis path-level reasoning tasks.
- 210 Script Generation tasks split between commercial giants (90 for Cadence Innovus, 88 for Synopsys ICC2), Engineering Change Orders (10 ECO), and Formal Verification (22 FM).
To evaluate full-flow implementation, the authors test the agents on 10 diverse circuit designs ranging from small control blocks (like a 176-cell Elevator FSM) to massive industrial-scale designs (like the 1.5-million-cell NVIDIA Deep Learning Accelerator, NVDLA-large).
Key Insights and Ideas
- Syntax is the Enemy, Not Concepts (Finding 1): An overwhelming 92% of script-generation failures were "near-misses." The LLMs correctly identified the right tool command but hallucinated or mismatched specific arguments and flags.
- Perception Outpaces Action (Finding 3): Models are excellent at reading and interpreting EDA reports (~90% accuracy), but their performance drops significantly (~50%) when asked to diagnose the root cause or formulate an optimization strategy based on those reports.
- The Power of Skill Augmentation (Finding 4): Standard Retrieval-Augmented Generation (RAG) baselines fail early in the design flow because general documentation is too coarse. Explicitly grounding agents with structured "skill packs" allows them to bypass fragile script generation and focus on high-level PPA (Power, Performance, Area) trade-offs.
- VLMs Excel at Spatial Reasoning: When given visual layouts as inputs, vision-language models (VLMs) enhanced by human skills achieved superior Worst Negative Slack (WNS) and Total Negative Slack (TNS) in macro placement, outperforming specialized analytical and reinforcement learning-based placers.
Key Results
The authors evaluated 11 state-of-the-art models (including GPT-5 variants, Claude 4.x, and Qwen3-VL).
- State-of-the-Art Baseline: GPT-5.5 emerged as the top performer, achieving 73.3% on root-cause analysis and 80.4% on foundational knowledge. However, even GPT-5.5 scored below 50% on raw script generation (45.5% for ICC2, 42.2% for Innovus), highlighting the difficulty of tool-centric execution.
- End-to-End Success: Driven by a GPT-5-mini backbone, the full PDAGENT framework successfully completed the entire RTL-to-GDSII physical design flow on industrial designs (TinyRISCV, AES-256, Ethernet MAC) using a commercial TSMC 28nm process, successfully closing timing and resolving all design rule check (DRC) violations.
Limitations and Future Work
While a major milestone, the authors note several limitations:
- Signoff Completeness: Layout-versus-Schematic (LVS) verification is currently excluded from the flow.
- Automation Gaps: Certain tasks that heavily rely on GUI-based interactions or manual engineer inspection (like complex DRC fixing) are not yet fully automated.
- Dataset Scale: The benchmark does not yet fully capture the extreme corner cases and heterogeneity of multi-billion-transistor, cutting-edge industrial chips.
Future Directions: The team plans to expand the benchmark to advanced nodes (16nm, 7nm) and mature nodes (65nm, 130nm), incorporate long-term memory and self-improvement modules so agents can learn from past failures, and integrate automated macro placement directly into the core framework loop.
Summary updated