Towards Efficient Agents: A Co-Design of Inference Architecture and System

Weizhe Lin, Hui-Ling Zhen, Shuai Yang, Xian Wang, Renxi Liu, Hanting Chen, Wangze Zhang, Chuansai Zhou, Yiming Li, Chen Chen 0077, Xing Li, Zhiyuan Yang, Xiaosong Li, Xianzhi Yu, Zhenhua Dong, Mingxuan Yuan, Yunhe WangarXiv 2025 · DOI · arXiv

Introduction

While Large Language Model (LLM) optimization typically focuses on increasing "tokens per second" (intra-step efficiency), autonomous agents present a different challenge. These agents operate in a continuous "Think–Act–Observe" loop, where the primary metric of success is "time-to-task-completion" rather than raw generation speed. In this context, a single reasoning error can trigger a cascade of failures, leading to dozens of unnecessary cycles and ultimately slowing down the system.

Huawei researchers identify an Efficiency Paradox: standard optimization techniques like INT8 quantization or aggressive context summarization, which typically benefit chatbots, can actually degrade agent performance. This degradation forces agents into expensive self-correction loops, negating any gains in raw throughput. To address this, the authors introduce AgentInfer, a unified framework that co-designs reasoning architectures with system-level scheduling and compression.

The AgentInfer Framework

AgentInfer decomposes the optimization problem into four synergistic modules designed to improve end-to-end efficiency while maintaining reasoning quality.

Overview of the AgentInfer framework and its four optimization modules.
  1. AgentCollab (Hierarchical Reasoning): This module utilizes a dual-model setup. A smaller, faster model handles routine tasks, while a larger model is reserved for high-level planning. A "Progress Check" mechanism monitors the small model; if it becomes stuck or fails to make progress, the task is automatically escalated to the large model.
  2. AgentCompress (Semantic Compression): Unlike standard summarization, this module asynchronously distills the agent's memory. It prioritizes preserving internal "reasoning memory" while pruning redundant environmental observations, keeping the context window lean without introducing cognitive ambiguity.
  3. AgentSched (Cache-Aware Scheduling): Standard schedulers often evict the KV-cache of long-running agents to accommodate shorter requests. AgentSched uses a "shadow-price" controller to balance prioritizing short jobs with preserving the memory of long-running agents, avoiding costly re-computation.
  4. AgentSAM (Speculative Decoding): This module employs a Suffix Automaton (SAM) to predict future tokens based on current and historical session patterns. To maintain low Time-to-First-Token (TTFT), the SAM is constructed in a background thread.

Key Insights: The Quantization Trap

One of the most significant findings is the "Quantization Trap." While moving from FP16 to INT8 quantization increases raw throughput by 45%, it results in a 30% drop in task success rate. This loss in accuracy forces the agent into extensive retry loops, increasing the average recovery time by 1000%. Consequently, the total time-to-solution increases by 70%, making the "faster" quantized model significantly slower in practice.

Similarly, the researchers observed a "Scissor Effect" in context summarization. While aggressive summarization reduces tokens per step, it can lead to a 3.5x increase in the number of turns required to solve a problem. These insights highlight that for autonomous agents, reasoning stability is a critical efficiency feature.

Results and Evaluation

The AgentInfer framework was evaluated on complex benchmarks including BrowseComp-zh (web browsing) and DeepDiver (iterative reasoning) using the openPangu model family.

  • End-to-End Speedup: AgentInfer achieved a 1.8x to 2.5x speedup in task completion time compared to native LLM agent baselines.
  • Token Efficiency: The framework reduced ineffective token consumption—tokens spent on failed paths or retries—by over 50%.
  • Accuracy and System Performance: AgentCollab recovered approximately 85% of the accuracy gap between small and large models. Additionally, AgentSched improved the average KV-cache hit rate from 63% to 72%, stabilizing performance during long-horizon tasks.

Conclusion

AgentInfer demonstrates that optimizing autonomous agents requires a shift from intra-step metrics to holistic, end-to-end system performance. By prioritizing reasoning stability and co-designing the inference stack, it is possible to achieve significant speedups and token savings without sacrificing the accuracy required for complex, multi-turn tasks. Future work may explore "Self-Evolution," where agents use distilled memories to fine-tune themselves, further reducing the need for model escalation.

Summary updated

Abstract

The rapid development of large language model (LLM)-based agents has unlocked new possibilities for autonomous multi-turn reasoning and tool-augmented decision-making. However, their real-world deployment is hindered by severe inefficiencies that arise not from isolated model inference, but from the systemic latency accumulated across reasoning loops, context growth, and heterogeneous tool interactions. This paper presents AgentInfer, a unified framework for end-to-end agent acceleration that bridges inference optimization and architectural design. We decompose the problem into four synergistic components: AgentCollab, a hierarchical dual-model reasoning framework that balances large- and small-model usage through dynamic role assignment; AgentSched, a cache-aware hybrid scheduler that minimizes latency under heterogeneous request patterns; AgentSAM, a suffix-automaton-based speculative decoding method that reuses multi-session semantic memory to achieve low-overhead inference acceleration; and AgentCompress, a semantic compression mechanism that asynchronously distills and reorganizes agent memory without disrupting ongoing reasoning. Together, these modules form a Self-Evolution Engine capable of sustaining efficiency and cognitive stability throughout long-horizon reasoning tasks. Experiments on the BrowseComp-zh and DeepDiver benchmarks demonstrate that through the synergistic collaboration of these methods, AgentInfer reduces ineffective token consumption by over 50%, achieving an overall 1.8-2.5 times speedup with preserved accuracy. These results underscore that optimizing for agentic task completion-rather than merely per-token throughput-is the key to building scalable, efficient, and self-improving intelligent systems.
BibTeX
@misc{lin2025towards,
  author = {Weizhe Lin and Hui-Ling Zhen and Shuai Yang and Xian Wang and Renxi Liu and Hanting Chen and Wangze Zhang and Chuansai Zhou and Yiming Li and Chen Chen and Xing Li and Zhiyuan Yang and Xiaosong Li and Xianzhi Yu and Zhenhua Dong and Mingxuan Yuan and Yunhe Wang},
  title = {{Towards Efficient Agents: A Co-Design of Inference Architecture and System}},
  year = {2025},
  eprint = {2512.18337},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18337},
}
Figure