Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in software engineering, yet their performance in the hardware-centric domain of embedded systems remains largely unexplored. This paper introduces EmbedAgent, a novel paradigm that simulates real-world roles—Programmer, Architect, and Integrator—to evaluate LLMs across the digital-physical divide. Accompanying this is EmbedBench, the first comprehensive benchmark for embedded development, featuring 126 cases across three hardware platforms (Arduino, ESP32, and Raspberry Pi Pico). Our evaluation of 10 mainstream LLMs reveals significant challenges: even state-of-the-art reasoning models like DeepSeek-R1 struggle with basic hardware logic and complex frameworks like ESP-IDF. We propose two enhancement strategies—Retrieval-Augmented Generation (RAG) and iterative compiler feedback—which significantly improve performance, highlighting the need for hardware-aware reasoning in future AI agents.
Introduction: The Digital-Physical Gap
While LLMs excel at generating Python scripts or web applications, embedded system development presents a unique set of constraints. It requires a "hardware-aware" mindset where code must interact precisely with pins, voltages, and physical components. Most existing benchmarks verify code through simple text output, which fails to capture real-world failures like incorrect pin mappings or neglected debouncing logic. To address this, we present a rigorous framework to evaluate the end-to-end process of embedded development.
The EmbedAgent Paradigm
To systematically test LLM capabilities, we define three distinct roles that mirror the professional development lifecycle:
- Embedded System Programmer: The model is provided with a circuit schematic and must write the corresponding firmware.
- Embedded System Architect: Starting from a high-level text description, the model must design the circuit (defining pin connections) and then write the code.
- Embedded System Integrator: The model must migrate existing code and schematics from one platform (e.g., Arduino) to another (e.g., ESP32 or Raspberry Pi Pico).
EmbedBench: A Comprehensive Benchmark
We developed EmbedBench to provide a standardized evaluation environment. It consists of 126 manually curated cases involving 9 common electronic components, such as RGB LEDs, 7-segment displays, and servos. The benchmark spans three diverse platforms: Arduino Uno (C++), ESP32 (ESP-IDF), and Raspberry Pi Pico (MicroPython).
To ensure scalable and objective verification, we utilize the Wokwi simulation platform. This allows our automated pipeline to "assemble" the virtual circuit and monitor the real-time state of components to verify if the hardware behaves as intended, rather than just checking for syntactical correctness.
Key Insights and Model Performance
Our experiments uncovered several surprising behaviors in modern LLMs:
- The "Wait" Proxy: Reasoning models like DeepSeek-R1 often use the word "wait" in their chain-of-thought when encountering hardware uncertainty. We found a direct correlation between the frequency of this term and the model's struggle with hardware logic.
- The Architect's Advantage: Interestingly, models sometimes perform better when designing the circuit themselves (Role II) than when following a provided schematic (Role I). This suggests they choose pin mappings that align with their internal logical paths.
- Overthinking Simple Logic: Reasoning models frequently "over-analyze" basic tasks. Instead of retrieving standard lookup tables for components like 7-segment displays, they often attempt to derive binary states from scratch, leading to avoidable errors.
- The Syntax Wall: While models handle MicroPython well, they struggle with the complex header requirements and API versions of ESP-IDF, leading to high failure rates during migration tasks.
Results and Enhancement Strategies
DeepSeek-R1 emerged as the top-performing model but still only achieved a 55.6% pass@1 rate on the Programmer task. Cross-platform migration proved even more difficult; the best model, Claude 3.7 Sonnet, reached only 29.4% accuracy when moving from Arduino to ESP32.
To mitigate these issues, we proposed and tested two strategies:
- R1-Retrieval: Using a general-purpose model (DeepSeek-V3) to provide domain-specific knowledge (like digit tables) to the reasoning model. This boosted DeepSeek-R1’s performance to 65.1%.
- R1-Compiler: An iterative loop where compiler error messages are fed back to the model. This reduced syntax errors on the ESP32 platform from 34.1% to 3.2% in just one iteration.
Conclusion
EmbedAgent and EmbedBench provide a critical foundation for evaluating AI in the physical world. Our findings suggest that while LLMs possess significant potential, bridging the digital-physical gap requires moving beyond pure reasoning toward systems that can effectively leverage domain knowledge and iterative feedback.
Summary updated