EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware

Minghui Zhao, Yiming Li, Sophie Zhu, Maxwell Khor, Lauren Ouyang, Hai "Helen" Li, Yiran Chen, Tingjun ChenarXiv 2026 · arXiv

TL;DR

EmbedGym is an interactive virtual hardware environment that allows AI coding agents to develop and debug embedded firmware without needing physical boards. By providing instruction-level emulation, analog circuit simulation, and a deterministic virtual clock, it enables agents to close the loop on hardware-specific bugs, achieving pass rates of up to 89.1% and allowing lightweight models to match frontier models at 30–70× lower cost.

What Problem Does This Paper Address?

In general software development, AI agents excel because they can run code and unit tests in a "closed loop"—they see the error, fix the code, and try again. Embedded firmware development breaks this loop. A program might compile perfectly but fail on real hardware due to incorrect pin polarities, bus protocol timing issues, or misconfigured sensor scaling.

Existing benchmarks for embedded AI either rely on static code analysis (which misses runtime bugs), use slow browser-based simulators that lack timing precision, or require physical hardware rigs that are impossible to scale. The authors identify a "compilation-to-runtime gap": even when firmware compiles 99.7% of the time, nearly 20% of those programs fail when actually executed on hardware.

Methods

The authors introduce EmbedGym, a high-fidelity simulation stack designed for agent interaction. It consists of three primary layers:

  1. Hardware Emulation Layer: Uses instruction-level emulators (like QEMU and Renode) to run the exact binary that would be flashed to a physical chip. It supports 41 peripheral models and includes an ngspice solver for analog circuits.
  2. Hardware CLI: A standardized, timestamped syntax (set, do, read, and assert) that allows agents to interact with the virtual environment. An agent can "press" a virtual button or "read" an LCD screen via text commands.
  3. Aligned Virtual Clock: Unlike simulators that run on "wall-clock" time (which suffers from host machine jitter), EmbedGym binds all actions to the CPU's internal clock. It also aligns the time origin ($t=0$) to the start of the user's application code, ensuring consistent behavior across different hardware architectures.
Figure 1: Overview of EmbedGym and EmbedGym-Bench: a coding agent iteratively writes, compiles, and tests embedded firmware through EmbedGym’s command-line interface against instruction-level MCU emulators, 41 peripheral models, an ngspice circuit solver, and virtual WiFi/BLE endpoints (§3), while EmbedGym-Bench evaluates the submitted binary across 598 tasks over 6 MCU-framework pairs using hidden, precisely timed test vectors (§4).

To build the accompanying benchmark, EmbedGym-Bench, the authors developed a five-stage multi-agent workflow. This pipeline automatically synthesizes peripheral models and tasks from datasheets and driver libraries, requiring only 4.6 minutes of human effort per task.

Datasets

The paper introduces EmbedGym-Bench, a cross-platform benchmark featuring:

  • 598 tasks across 6 different Microcontroller Unit (MCU)-framework pairs (including ATmega328P, ESP32-S3, and nRF52840).
  • 117 unique task specifications spanning domains like wireless networking (WiFi/BLE), sensing, and UI.
  • Hidden timed test vectors that grade submissions based on actual hardware states rather than just compilation or serial logs.

Key Insights and Ideas

  • The Failure of Passive Feedback: Simply giving an agent serial logs (passive feedback) is insufficient. The authors found that passive logs often mislead agents, causing them to break previously working code because static idle outputs mask input-dependent bugs.
  • Active Stimulus is the Key: The biggest performance gains came from "Active Stimulus" (Level 3 interaction), where agents could perturb sensor values and inspect internal states. This allowed agents to catch bugs like inverted logic or wrong register bitmasks that are invisible at rest.
  • Sim-to-Real Fidelity: The environment is remarkably accurate. In "Hardware-in-the-Loop" testing, the authors found a 98.3% agreement between EmbedGym’s simulation results and actual physical hardware boards.
  • Democratizing Frontier Performance: One of the most striking findings is that interactive hardware access allows lightweight models (like GPT-4o-mini, referred to in the paper's evaluation as GPT-6 Luna) to match the performance of much larger frontier models that lack such access, but at a fraction of the token cost.
Comparison of agent pass rates across different feedback levels.

Key Results

  • Pass Rate Boost: Interactive testing in EmbedGym increased agent pass rates by +4.9% to +14.7% across various models, reaching up to +31% on specific hardware targets.
  • Overall Accuracy: The best-performing agents achieved an overall pass rate of 89.1% across all 598 tasks.
  • Efficiency: The automated scaling workflow reduced the human effort required to create new tasks from hours of manual coding to just minutes of oversight.
  • Cost-Effectiveness: Agents using EmbedGym outperformed prior frameworks while running at 30–70× lower cost by enabling more efficient debugging cycles.

Limitations and Future Work

While EmbedGym is highly accurate, the authors noted a rare "sim-to-real" mismatch involving a specific compass sensor's register-locking behavior that wasn't perfectly replicated in the model.

Future work could expand the peripheral library further and explore more complex multi-MCU systems. Additionally, while the current focus is on functional correctness, extending the environment to profile power consumption or perform security vulnerability research (like side-channel analysis) represents a promising next step for the platform.

Summary updated

Abstract

Modern coding agents perform well on general software benchmarks because they can run code and unit tests in a closed loop. Embedded firmware development, however, breaks this loop: a program that compiles can still fail on real hardware due to pin polarity, bus protocol, timing, or networking bugs that an agent at a text terminal cannot observe. Existing embedded coding agents either skip hardware execution, drive GPIO-only simulators with multi-second delays, or test on physical boards in a static environment. In this paper, we present EmbedGym, an interactive virtual hardware environment that lets coding agents develop and debug embedded firmware without physical hardware. EmbedGym uses instruction-level microcontroller unit (MCU) emulators to execute the same binary that would be flashed onto a physical board, connects them to 41 peripheral models, an analog circuit simulator, and virtual WiFi/Bluetooth Low Energy (BLE) endpoints, and exposes an interface for agents to change sensor inputs, press buttons, and inspect hardware states. On top of it, we build EmbedGym-Bench, which consists of 598 tasks across 6 MCU-framework pairs graded by hidden timed test vectors, using a five-stage multi-agent workflow that synthesizes peripheral models and tasks from datasheets and driver libraries. We expose EmbedGym to a coding agent at three levels and evaluate eight models over more than 14K trajectories. Even when compilation succeeds 99.7% of the time, 19.9% of compiled firmwares fail at runtime (up to 73% on individual targets). In contrast, interactive hardware testing in EmbedGym increases pass rates across models by +4.9% to +14.7% (up to +31% on individual targets). Our results also show that an agent with EmbedGym outperforms prior embedded agent frameworks, and lightweight models with EmbedGym match frontier models without it at 30-70x lower cost.
BibTeX
@misc{zhao2026embedgym,
  author = {Minghui Zhao and Yiming Li and Sophie Zhu and Maxwell Khor and Lauren Ouyang and Hai "Helen" Li and Yiran Chen and Tingjun Chen},
  title = {{EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware}},
  year = {2026},
  eprint = {2611.21845},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21845},
}
Figure