TL;DR
RealEmbedBench is a new benchmark specifically designed to evaluate how well Large Language Models (LLMs) handle the complexities of application-layer embedded software. The study reveals a significant "implementation gap": while models can often model high-level logic like state machines, they frequently struggle to translate those designs into correct, project-specific executable code.
What Problem Does This Paper Address?
While LLMs have become remarkably good at general-purpose coding, the embedded systems world remains a "frontier" with unique challenges. Existing benchmarks tend to fall into two camps: general software (like HumanEval) or low-level hardware-in-the-loop tasks (like EmbedAgent).
This leaves a massive gap in the application layer—the place where developers build complex business logic, manage state machines, and migrate code across different Real-Time Operating Systems (RTOS). In this layer, code isn't just about logic; it’s constrained by strict resource limits, real-time requirements, and highly specific project contexts. The authors created RealEmbedBench to see if LLMs can actually handle these "real-world" engineering hurdles.
Methods
The researchers didn't just guess what developers find hard; they started with a preliminary study of 125 recurring "pain points" from StackOverflow and Reddit, validated by interviews with 12 professional developers.
Based on this, they built a multi-stage construction pipeline to create three task categories:
- Code Generation & Completion: Generating or filling in code within complex project structures.
- State Machine Modeling & Implementation: Moving between natural language requirements, PlantUML models, and C code.
- Cross-OS Code Translation: Porting code between different RTOSes (e.g., FreeRTOS to Zephyr).
The pipeline (shown below) used a mix of automated static analysis (Clang-based), LLM-assisted generation (using DeepSeek-v3.2), and rigorous manual review by human experts to ensure correctness.
Datasets
The authors introduced RealEmbedBench, which comprises 5,142 samples derived from 15 large, active open-source embedded projects (including well-known ones like Ardupilot and Marlin).
The benchmark includes:
- 1,426 code generation samples.
- 3,026 code completion samples.
- 80 tri-modal state machine samples (Requirements + PlantUML + Code).
- 610 cross-OS translation pairs covering 6 different operating systems and 26 translation directions.
Key Insights and Ideas
- The "Local Window" is Critical: For code generation, the most important piece of context isn't global data or type definitions—it's the immediate "nearby" code. Without this local context, models often break the established style or control flow of the file.
- Modeling vs. Implementing: Models are surprisingly good at "drawing the map" (creating PlantUML diagrams from requirements) but struggle to "drive the car" (writing the actual C code). They often hallucinate APIs or simplify complex transition logic that was explicitly required.
- Translation Asymmetry: Porting code is not a symmetric task. For example, models found translating from RT-Thread to Harmony relatively easy, but the reverse (Harmony to RT-Thread) was a "black hole" for almost every model tested.
- API Hallucination: A major failure mode in state machines was the invention of plausible-sounding but non-existent helper functions, rather than using the project's actual member variables or established interfaces.
Key Results
The researchers evaluated six major models, including DeepSeek-V3.2, GPT-5.2, and Gemini-3.1-Pro.
- Overall Champion: Gemini-3.1-Pro achieved the best average performance across most tasks, particularly in code generation (scoring 0.7076).
- Translation Specialist: GLM-4.7 proved strongest in cross-OS translation, likely due to better exposure to diverse RTOS-specific training data.
- The Harmony Hurdle: The translation from Harmony to RT-Thread was the most difficult task in the entire benchmark, with an average pass rate of only 0.1667 across all models.
- Context Sensitivity: Removing all project context (the "NoCtx" setting) caused performance to crater; for instance, Gemini’s score dropped from 0.7076 to 0.4149.
Limitations and Future Work
The authors acknowledge that while RealEmbedBench is a major step forward, there is still a clear gap between structural understanding and faithful code implementation in LLMs.
Future Work Directions: * Better Training Data: LLM trainers need to include more data that explicitly connects high-level design semantics (like state machine skeletons) with concrete implementation details. * Implicit Constraints: Models need a deeper understanding of the "hidden" behavioral constraints of different RTOSes, such as timing semantics and memory handling, which are often missed during translation. * Expanding Subdomains: Future versions of the benchmark could expand into even more specialized embedded fields like medical devices or aerospace-specific protocols.
Summary updated