EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

Zhengxiang Huang, Yujie Sun, Shengheng Chen, Zhaode Wang, Chaoyue Niu, Zeyu Zhao, Chengfei Lv, Fan Wu, Guihai ChenarXiv 2026 · arXiv

TL;DR

EngineAgent is a multi-agent system designed to automate the complex process of porting new Large Language Models (LLMs) to mobile inference engines. By explicitly modeling engine hierarchies and pre-validating hardware-specific debugging workflows, it achieves a 75% success rate in model enablement, more than doubling the performance of general-purpose coding agents.

What Problem Does This Paper Address?

Deploying the latest LLMs on mobile devices is a "recurring and costly" engineering nightmare. On one side, model architectures are evolving at breakneck speed—sometimes releasing new versions just weeks apart. On the other side, mobile hardware is a fragmented landscape of ARM CPUs, various GPUs (Vulkan, OpenCL), and specialized NPUs (Qualcomm, MediaTek, Apple).

Inference engines like llama.cpp or MNN act as the bridge between these worlds. Supporting a new model isn't just about changing a few lines of code; it requires coordinated modifications across a complex hierarchy:

  1. Input Level: Handling new tokenizers or multimodal patchification.
  2. Graph Level: Mapping new model layers to the engine’s internal representation.
  3. Operator Level: Writing or adapting hardware-specific kernels (e.g., assembly for CPUs or shaders for GPUs).

Existing AI coding agents (like SWE-agent or Codex) usually fail here because they treat codebases as flat collections of files. They miss "cross-layer" dependencies—like adding a new operator definition but forgetting to implement its hardware-specific kernel—and they struggle with the "host-device" debugging loop, where a failure on a phone might actually be caused by a bug in the host-side model converter.

Methods

The authors introduce EngineAgent, which tackles these challenges through a two-stage process:

1. Model-Agnostic Pre-Enablement

Before a new model even arrives, EngineAgent prepares by "learning" the engine.

  • Hierarchy-Aware Engine Index: It constructs a 9-layer index (L1–L9) that maps code entities to their functional roles in the inference stack (e.g., L1 for tokenization, L7 for attention mechanisms). This allows the agent to understand the semantic role of code rather than just its syntactic structure.
  • Hardware-Aware Debug Skills: The system uses already supported models to validate the build, deployment, and testing workflows for specific backends (CPU, GPU, NPU). This ensures that when a new model fails, the agent has a stable, pre-validated pipeline to use for debugging.

2. Model-Specific Multi-Agent Loop

When a new model is released, three specialized agents work together:

  • Builder: Performs a "difference analysis" between the new model and existing ones, then implements changes layer-by-layer guided by the hierarchy index.
  • Auditor: Acts as a "red-team" verifier, running unit tests for each layer and ensuring the implementation is complete before moving on.
  • Debugger: If a test fails on the device, the debugger uses the pre-constructed "skills" to trace the error back to the specific layer or backend implementation.
Figure 1. Position and workflow of EngineAgent: agentically mapping rapidly evolving LLM architectures onto heterogeneous mobile hardware for inference engines.

Datasets

The authors developed a benchmark of 12 model-enablement tasks to evaluate the system.

  • Models: Qwen3.5-2B, Gemma4-E2B-it, and MiniCPM-V 4.6.
  • Engine-Backend Pairs: MNN + CPU, llama.cpp + Vulkan, MLC-LLM + OpenCL, and ExecuTorch + QNN (NPU).
  • Evaluation Criteria: Success is strictly defined as passing host compilation, successful on-device execution, and "logit agreement"—ensuring the mobile output matches a reference PyTorch implementation within a tiny numerical tolerance.

Key Insights and Ideas

  • Hierarchy over Structure: The "aha moment" is that for systems programming, understanding the abstraction layers of an engine is more important than understanding the file tree. By mapping code to a 9-layer hierarchy, the agent can identify missing pieces that general agents overlook.
  • Stable Verification Paths: Debugging a new model on hardware is noisy. By pre-validating the "plumbing" (build scripts, ADB deployment, log collection) using known-good models, EngineAgent removes a massive source of uncertainty for the AI.
  • Agent Specialization: Splitting the task into a Builder, Auditor, and Debugger prevents the "premature exit" problem where an agent assumes its code works just because it compiled on the host.
Figure 3. Model enablement workloads: mapping new model features to engine abstraction layers and then perform host- device build-verify-debug loop.

Key Results

  • Superior Success Rate: EngineAgent achieved a 75% end-to-end success rate (9/12 tasks). In comparison, the strongest baseline (Codex) only managed 33.3%, and most other agents (like SWE-agent) failed nearly every task.
  • Efficiency: Using the same underlying LLM (DeepSeek-V4-Flash), EngineAgent was faster (3.82 hours vs. ~4.8 hours) and cheaper ($1.53 vs. ~$2.20 in API costs) per task than the baselines.
  • Ablation Proof: Removing the hierarchy index caused the success rate to plummet to 25%, proving that "system awareness" is the secret sauce.

Limitations and Future Work

While highly effective, EngineAgent still failed on 3 out of 12 tasks. These failures were primarily in complex graph-level logic and highly specialized hardware kernel implementations.

The authors suggest that future work could involve:

  • Integrating more advanced automated kernel generation (like Triton or MoKA).
  • Expanding the hierarchy index to support even more diverse engine architectures.
  • Improving the "Auditor" agent's ability to catch subtle numerical drifts earlier in the pipeline.

Summary updated

Abstract

Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.
BibTeX
@misc{huang2026engineagent,
  author = {Zhengxiang Huang and Yujie Sun and Shengheng Chen and Zhaode Wang and Chaoyue Niu and Zeyu Zhao and Chengfei Lv and Fan Wu and Guihai Chen},
  title = {{EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities}},
  year = {2026},
  eprint = {2611.14356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14356},
}
Figure