Agentic Performance at the Edge: Insights from Benchmarking

Shiqiang Wang, Herbert Woisetschläger

TL;DR

Deploying AI agents at the edge requires squeezing complex reasoning into models under 8 billion parameters. This paper benchmarks compact open-source models (Qwen, Phi, Mistral) in tool-driven environments, revealing that agentic performance does not scale smoothly with parameter count. Instead, successful edge deployment depends on a nuanced combination of task domain, model architecture (like coder-tuning), and targeted error-handling strategies.

What Problem Does This Paper Address?

Large Language Models (LLMs) are evolving from simple text generators into autonomous "agents" capable of planning, reasoning, and calling external tools to solve multi-step problems. Running these agents locally at the edge—such as on on-premise gateways, factory controllers, or local servers—is highly desirable for data privacy, offline resilience, and eliminating cloud latency.

However, edge hardware imposes strict compute, memory, and power constraints. While recent compact models (e.g., Phi-3, TinyLlama) show promise for static question-answering, interactive tool-grounded tasks are significantly harder. This paper addresses a critical systems tension: How much agentic capability is lost under strict edge constraints, and how should engineers select and deploy small models when "just use a bigger model" isn't an option?

Methods

The authors design a controlled evaluation framework to isolate model capabilities from prompt engineering noise. Unlike leaderboards that optimize prompts per model, this study holds the entire agent infrastructure fixed:

  • Fixed Protocol: A strict iterative loop (based on the ReAct framework) allowing one tool call per turn, capped at a conservative 12-step compute budget.
  • Controlled Decomposition: The researchers systematically isolate three variables: model size, model generation (older vs. newer versions), and model variant (general-purpose vs. coder-oriented).
  • Failure-Mode Analysis: Instead of just tracking accuracy, the framework categorizes failures into semantic errors (the model completes the task but gives the wrong answer) and execution errors (the model breaks the tool workflow or hits the step limit).

Datasets

The framework is evaluated using the scorable subset of ITBench, a benchmark focused on real-world IT automation and incident diagnosis. It covers two distinct operational domains:

  1. FinOps (Financial Operations): 25 tasks requiring the agent to analyze structured cloud billing data to detect and attribute cost anomalies.
  2. SRE (Site Reliability Engineering): 35 tasks requiring multi-source log aggregation, topology reasoning, and root-cause localization in distributed systems.

Key Insights and Ideas

  • Scaling is Non-Steady: At the edge, bigger isn't always better. For example, Qwen2.5-3B actually outperforms the larger Qwen2.5-7B in overall accuracy.
  • Domain Asymmetry Dominates Size: The inherent structure of a task matters more than parameter count. FinOps tasks (structured data analysis) are universally easier for small models, while SRE tasks (heterogeneous log and topology analysis) act as a severe bottleneck across the board.
  • The Coder Threshold Effect: Tuning models on code provides a massive boost to tool-use reliability, but only after a model reaches a baseline capacity. For instance, Qwen's coder variant yields huge gains at the 7B size but actually degrades performance at the 3B size.
  • Ecosystems Fail Differently: Qwen models excel at tool execution but suffer from semantic failures (picking the wrong final answer). Conversely, Phi and Mistral models are prone to execution failures (getting lost in the tool loop). This implies Qwen is ideal when a human-in-the-loop verifies the final output, while Phi/Mistral are better suited for early automated fallback triggers.

Key Results

  • The Efficiency Frontier: Qwen Coder 7B emerged as a standout choice for edge deployment. It achieved the exact same peak accuracy (33.9%) as its 14B and 32B counterparts but delivered a 4.3× latency reduction, completing tasks in an average of 1.45 seconds compared to 6.22 seconds for the 32B model.
Figure 1: Size-view accuracy by family (each subfigure uses the family’s own ascending size order).
  • Generation Gaps: Upgrading generations doesn't guarantee success. While Phi4 Mini (3.8B) achieved a strong 28.6% accuracy, its predecessor Phi 3.5 (3.8B) suffered a total performance collapse (0% accuracy) under the exact same protocol constraints.
Figure 2: Generation-view accuracy by family.
  • The SRE Bottleneck: While top models achieved 56–63% accuracy on FinOps tasks, performance on SRE tasks maxed out at a dismal 6–10%, highlighting that multi-source evidence gathering remains an open challenge for edge-sized models.

Limitations and Future Work

The primary limitation is that the study evaluates a single benchmark framework (ITBench) across three open-source model families. The authors note that future work should expand this methodology to other edge-relevant domains like SecOps (Security Operations). Additionally, they suggest exploring hybrid edge-cloud split inference (selectively offloading hard steps to the cloud) and investigating how the specific design of tool interfaces can be optimized to reduce execution errors in smaller models.

Summary updated

Abstract

Agentic artificial intelligence (AI) is a natural fit for Internet of Things (IoT) and edge systems, but edge deployments are often constrained to models around 8 billion parameters or smaller. An important question is: How much agentic-task quality is lost when model size is constrained by memory, power, and latency budgets? To address this question, in this paper, we provide an initial empirical study considering edge-focused model scaling, general-purpose versus coder-oriented model effects, and tool-enabled execution under a fixed protocol. We introduce a domain-conditioned evaluation methodology, an implementation-grounded analysis of model-tool interactions, practical guidance for model selection under constraints, and an analysis of failure modes that reveals distinct semantic versus execution failure patterns across model families. Our core finding is that edge-agent quality is not a simple function of parameter count. Robust deployment depends on the joint design of model choice and tool workflow. Domain-conditioned analysis reveals Pareto fronts in the accuracy-latency space that can guide strategy selection based on operational priorities.
BibTeX
@inproceedings{wang2026agentic,
  author = {Shiqiang Wang and Herbert Woisetschläger},
  title = {{Agentic Performance at the Edge: Insights from Benchmarking}},
  booktitle = {MobiSys Workshops},
  year = {2026},
  doi = {10.1145/3812836.3814741},
}

Loading related content...

Figure