TL;DR
EmbedGym is an interactive virtual hardware environment that allows AI coding agents to develop and debug embedded firmware without needing physical boards. By providing instruction-level emulation, analog circuit simulation, and a deterministic virtual clock, it enables agents to close the loop on hardware-specific bugs, achieving pass rates of up to 89.1% and allowing lightweight models to match frontier models at 30–70× lower cost.
What Problem Does This Paper Address?
In general software development, AI agents excel because they can run code and unit tests in a "closed loop"—they see the error, fix the code, and try again. Embedded firmware development breaks this loop. A program might compile perfectly but fail on real hardware due to incorrect pin polarities, bus protocol timing issues, or misconfigured sensor scaling.
Existing benchmarks for embedded AI either rely on static code analysis (which misses runtime bugs), use slow browser-based simulators that lack timing precision, or require physical hardware rigs that are impossible to scale. The authors identify a "compilation-to-runtime gap": even when firmware compiles 99.7% of the time, nearly 20% of those programs fail when actually executed on hardware.
Methods
The authors introduce EmbedGym, a high-fidelity simulation stack designed for agent interaction. It consists of three primary layers:
- Hardware Emulation Layer: Uses instruction-level emulators (like QEMU and Renode) to run the exact binary that would be flashed to a physical chip. It supports 41 peripheral models and includes an
ngspicesolver for analog circuits. - Hardware CLI: A standardized, timestamped syntax (
set,do,read, andassert) that allows agents to interact with the virtual environment. An agent can "press" a virtual button or "read" an LCD screen via text commands. - Aligned Virtual Clock: Unlike simulators that run on "wall-clock" time (which suffers from host machine jitter), EmbedGym binds all actions to the CPU's internal clock. It also aligns the time origin ($t=0$) to the start of the user's application code, ensuring consistent behavior across different hardware architectures.
To build the accompanying benchmark, EmbedGym-Bench, the authors developed a five-stage multi-agent workflow. This pipeline automatically synthesizes peripheral models and tasks from datasheets and driver libraries, requiring only 4.6 minutes of human effort per task.
Datasets
The paper introduces EmbedGym-Bench, a cross-platform benchmark featuring:
- 598 tasks across 6 different Microcontroller Unit (MCU)-framework pairs (including ATmega328P, ESP32-S3, and nRF52840).
- 117 unique task specifications spanning domains like wireless networking (WiFi/BLE), sensing, and UI.
- Hidden timed test vectors that grade submissions based on actual hardware states rather than just compilation or serial logs.
Key Insights and Ideas
- The Failure of Passive Feedback: Simply giving an agent serial logs (passive feedback) is insufficient. The authors found that passive logs often mislead agents, causing them to break previously working code because static idle outputs mask input-dependent bugs.
- Active Stimulus is the Key: The biggest performance gains came from "Active Stimulus" (Level 3 interaction), where agents could perturb sensor values and inspect internal states. This allowed agents to catch bugs like inverted logic or wrong register bitmasks that are invisible at rest.
- Sim-to-Real Fidelity: The environment is remarkably accurate. In "Hardware-in-the-Loop" testing, the authors found a 98.3% agreement between EmbedGym’s simulation results and actual physical hardware boards.
- Democratizing Frontier Performance: One of the most striking findings is that interactive hardware access allows lightweight models (like GPT-4o-mini, referred to in the paper's evaluation as GPT-6 Luna) to match the performance of much larger frontier models that lack such access, but at a fraction of the token cost.
Key Results
- Pass Rate Boost: Interactive testing in EmbedGym increased agent pass rates by +4.9% to +14.7% across various models, reaching up to +31% on specific hardware targets.
- Overall Accuracy: The best-performing agents achieved an overall pass rate of 89.1% across all 598 tasks.
- Efficiency: The automated scaling workflow reduced the human effort required to create new tasks from hours of manual coding to just minutes of oversight.
- Cost-Effectiveness: Agents using EmbedGym outperformed prior frameworks while running at 30–70× lower cost by enabling more efficient debugging cycles.
Limitations and Future Work
While EmbedGym is highly accurate, the authors noted a rare "sim-to-real" mismatch involving a specific compass sensor's register-locking behavior that wasn't perfectly replicated in the model.
Future work could expand the peripheral library further and explore more complex multi-MCU systems. Additionally, while the current focus is on functional correctness, extending the environment to profile power consumption or perform security vulnerability research (like side-channel analysis) represents a promising next step for the platform.
Summary updated