EmbedAgent: Benchmarking Large Language Models in Embedded System Development

Ruiyang Xu, Jialun Cao, Mingyuan Wu, Wenliang Zhong, Yaojie Lu, Ben He, Xianpei Han, Shing-Chi Cheung, Le Sun

Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in software engineering, yet their performance in the hardware-centric domain of embedded systems remains largely unexplored. This paper introduces EmbedAgent, a novel paradigm that simulates real-world roles—Programmer, Architect, and Integrator—to evaluate LLMs across the digital-physical divide. Accompanying this is EmbedBench, the first comprehensive benchmark for embedded development, featuring 126 cases across three hardware platforms (Arduino, ESP32, and Raspberry Pi Pico). Our evaluation of 10 mainstream LLMs reveals significant challenges: even state-of-the-art reasoning models like DeepSeek-R1 struggle with basic hardware logic and complex frameworks like ESP-IDF. We propose two enhancement strategies—Retrieval-Augmented Generation (RAG) and iterative compiler feedback—which significantly improve performance, highlighting the need for hardware-aware reasoning in future AI agents.

Introduction: The Digital-Physical Gap

While LLMs excel at generating Python scripts or web applications, embedded system development presents a unique set of constraints. It requires a "hardware-aware" mindset where code must interact precisely with pins, voltages, and physical components. Most existing benchmarks verify code through simple text output, which fails to capture real-world failures like incorrect pin mappings or neglected debouncing logic. To address this, we present a rigorous framework to evaluate the end-to-end process of embedded development.

The EmbedAgent Paradigm

To systematically test LLM capabilities, we define three distinct roles that mirror the professional development lifecycle:

  1. Embedded System Programmer: The model is provided with a circuit schematic and must write the corresponding firmware.
  2. Embedded System Architect: Starting from a high-level text description, the model must design the circuit (defining pin connections) and then write the code.
  3. Embedded System Integrator: The model must migrate existing code and schematics from one platform (e.g., Arduino) to another (e.g., ESP32 or Raspberry Pi Pico).
The EmbedAgent paradigm defines three roles: Programmer, Architect, and Integrator to test different facets of embedded development.

EmbedBench: A Comprehensive Benchmark

We developed EmbedBench to provide a standardized evaluation environment. It consists of 126 manually curated cases involving 9 common electronic components, such as RGB LEDs, 7-segment displays, and servos. The benchmark spans three diverse platforms: Arduino Uno (C++), ESP32 (ESP-IDF), and Raspberry Pi Pico (MicroPython).

To ensure scalable and objective verification, we utilize the Wokwi simulation platform. This allows our automated pipeline to "assemble" the virtual circuit and monitor the real-time state of components to verify if the hardware behaves as intended, rather than just checking for syntactical correctness.

The evaluation pipeline bridges the gap between digital code and physical behavior by simulating the compilation and circuit execution.

Key Insights and Model Performance

Our experiments uncovered several surprising behaviors in modern LLMs:

  • The "Wait" Proxy: Reasoning models like DeepSeek-R1 often use the word "wait" in their chain-of-thought when encountering hardware uncertainty. We found a direct correlation between the frequency of this term and the model's struggle with hardware logic.
  • The Architect's Advantage: Interestingly, models sometimes perform better when designing the circuit themselves (Role II) than when following a provided schematic (Role I). This suggests they choose pin mappings that align with their internal logical paths.
  • Overthinking Simple Logic: Reasoning models frequently "over-analyze" basic tasks. Instead of retrieving standard lookup tables for components like 7-segment displays, they often attempt to derive binary states from scratch, leading to avoidable errors.
  • The Syntax Wall: While models handle MicroPython well, they struggle with the complex header requirements and API versions of ESP-IDF, leading to high failure rates during migration tasks.

Results and Enhancement Strategies

DeepSeek-R1 emerged as the top-performing model but still only achieved a 55.6% pass@1 rate on the Programmer task. Cross-platform migration proved even more difficult; the best model, Claude 3.7 Sonnet, reached only 29.4% accuracy when moving from Arduino to ESP32.

To mitigate these issues, we proposed and tested two strategies:

  1. R1-Retrieval: Using a general-purpose model (DeepSeek-V3) to provide domain-specific knowledge (like digit tables) to the reasoning model. This boosted DeepSeek-R1’s performance to 65.1%.
  2. R1-Compiler: An iterative loop where compiler error messages are fed back to the model. This reduced syntax errors on the ESP32 platform from 34.1% to 3.2% in just one iteration.

Conclusion

EmbedAgent and EmbedBench provide a critical foundation for evaluating AI in the physical world. Our findings suggest that while LLMs possess significant potential, bridging the digital-physical gap requires moving beyond pure reasoning toward systems that can effectively leverage domain knowledge and iterative feedback.

Summary updated

Abstract

Large Language Models (LLMs) have shown promise in various tasks, yet few benchmarks assess their capabilities in embedded system development.In this paper, we introduce EmbedAgent, a paradigm designed to simulate real-world roles in embedded system development, such as Embedded System Programmer, Architect, and Integrator. This paradigm enables LLMs to be tested in tasks that bridge the gap between digital and physical systems, allowing for a more comprehensive assessment of their capabilities. To evaluate LLMs on these tasks, we propose Embedbench, the first comprehensive benchmark for embedded system programming, circuit design, and cross-platform migration.Embedbench consists of 126 cases, covering 9 electronic components across 3 hardware platforms. Through extensive experiments on 10 mainstream LLMs, we uncover several key findings. Surprisingly, despite the simplicity of the cases, DeepSeek-R1 achieves only a 55.6% pass@1 rate when provided with schematic information, and 50.0% when tasked with generating the schematics itself. In the cross-platform migration tasks, LLMs show relatively strong performance with MicroPython on the Raspberry Pi Pico (with the top model achieving 73.8% pass@1), but perform poorly on ESP-IDF, where the best model reaches only 29.4% [email protected], we observe that general-purpose chat LLMs like DeepSeek-V3 often fail to utilize relevant pre-trained knowledge in this domain, while reasoning LLMs tend to overthink and overlook efficient knowledge during pretraining. Based on these insights, we propose two strategies: retrieval augmented generation and compiler feedback-to enhance LLM performance. These strategies result in significant improvements, with Deepseek-R1 reaching a 65.1% pass@1 with correct schematics, and 53.1% without. Additionally, the accuracy of the Arduino to ESP32 migration task improves from 21.4% to 27.8%.
BibTeX
@misc{xu2025embedagent,
  author = {Ruiyang Xu and Jialun Cao and Mingyuan Wu and Wenliang Zhong and Yaojie Lu and Ben He and Xianpei Han and Shing-Chi Cheung and Le Sun},
  title = {{EmbedAgent: Benchmarking Large Language Models in Embedded System Development}},
  year = {2025},
  eprint = {2506.11003},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.11003},
}
Figure