TL;DR
EngineAgent is a multi-agent system designed to automate the complex process of porting new Large Language Models (LLMs) to mobile inference engines. By explicitly modeling engine hierarchies and pre-validating hardware-specific debugging workflows, it achieves a 75% success rate in model enablement, more than doubling the performance of general-purpose coding agents.
What Problem Does This Paper Address?
Deploying the latest LLMs on mobile devices is a "recurring and costly" engineering nightmare. On one side, model architectures are evolving at breakneck speed—sometimes releasing new versions just weeks apart. On the other side, mobile hardware is a fragmented landscape of ARM CPUs, various GPUs (Vulkan, OpenCL), and specialized NPUs (Qualcomm, MediaTek, Apple).
Inference engines like llama.cpp or MNN act as the bridge between these worlds. Supporting a new model isn't just about changing a few lines of code; it requires coordinated modifications across a complex hierarchy:
- Input Level: Handling new tokenizers or multimodal patchification.
- Graph Level: Mapping new model layers to the engine’s internal representation.
- Operator Level: Writing or adapting hardware-specific kernels (e.g., assembly for CPUs or shaders for GPUs).
Existing AI coding agents (like SWE-agent or Codex) usually fail here because they treat codebases as flat collections of files. They miss "cross-layer" dependencies—like adding a new operator definition but forgetting to implement its hardware-specific kernel—and they struggle with the "host-device" debugging loop, where a failure on a phone might actually be caused by a bug in the host-side model converter.
Methods
The authors introduce EngineAgent, which tackles these challenges through a two-stage process:
1. Model-Agnostic Pre-Enablement
Before a new model even arrives, EngineAgent prepares by "learning" the engine.
- Hierarchy-Aware Engine Index: It constructs a 9-layer index (L1–L9) that maps code entities to their functional roles in the inference stack (e.g., L1 for tokenization, L7 for attention mechanisms). This allows the agent to understand the semantic role of code rather than just its syntactic structure.
- Hardware-Aware Debug Skills: The system uses already supported models to validate the build, deployment, and testing workflows for specific backends (CPU, GPU, NPU). This ensures that when a new model fails, the agent has a stable, pre-validated pipeline to use for debugging.
2. Model-Specific Multi-Agent Loop
When a new model is released, three specialized agents work together:
- Builder: Performs a "difference analysis" between the new model and existing ones, then implements changes layer-by-layer guided by the hierarchy index.
- Auditor: Acts as a "red-team" verifier, running unit tests for each layer and ensuring the implementation is complete before moving on.
- Debugger: If a test fails on the device, the debugger uses the pre-constructed "skills" to trace the error back to the specific layer or backend implementation.
Datasets
The authors developed a benchmark of 12 model-enablement tasks to evaluate the system.
- Models: Qwen3.5-2B, Gemma4-E2B-it, and MiniCPM-V 4.6.
- Engine-Backend Pairs: MNN + CPU, llama.cpp + Vulkan, MLC-LLM + OpenCL, and ExecuTorch + QNN (NPU).
- Evaluation Criteria: Success is strictly defined as passing host compilation, successful on-device execution, and "logit agreement"—ensuring the mobile output matches a reference PyTorch implementation within a tiny numerical tolerance.
Key Insights and Ideas
- Hierarchy over Structure: The "aha moment" is that for systems programming, understanding the abstraction layers of an engine is more important than understanding the file tree. By mapping code to a 9-layer hierarchy, the agent can identify missing pieces that general agents overlook.
- Stable Verification Paths: Debugging a new model on hardware is noisy. By pre-validating the "plumbing" (build scripts, ADB deployment, log collection) using known-good models, EngineAgent removes a massive source of uncertainty for the AI.
- Agent Specialization: Splitting the task into a Builder, Auditor, and Debugger prevents the "premature exit" problem where an agent assumes its code works just because it compiled on the host.
Key Results
- Superior Success Rate: EngineAgent achieved a 75% end-to-end success rate (9/12 tasks). In comparison, the strongest baseline (Codex) only managed 33.3%, and most other agents (like SWE-agent) failed nearly every task.
- Efficiency: Using the same underlying LLM (DeepSeek-V4-Flash), EngineAgent was faster (3.82 hours vs. ~4.8 hours) and cheaper ($1.53 vs. ~$2.20 in API costs) per task than the baselines.
- Ablation Proof: Removing the hierarchy index caused the success rate to plummet to 25%, proving that "system awareness" is the secret sauce.
Limitations and Future Work
While highly effective, EngineAgent still failed on 3 out of 12 tasks. These failures were primarily in complex graph-level logic and highly specialized hardware kernel implementations.
The authors suggest that future work could involve:
- Integrating more advanced automated kernel generation (like Triton or MoKA).
- Expanding the hierarchy index to support even more diverse engine architectures.
- Improving the "Auditor" agent's ability to catch subtle numerical drifts earlier in the pipeline.
Summary updated