Introduction
While Large Language Model (LLM) optimization typically focuses on increasing "tokens per second" (intra-step efficiency), autonomous agents present a different challenge. These agents operate in a continuous "Think–Act–Observe" loop, where the primary metric of success is "time-to-task-completion" rather than raw generation speed. In this context, a single reasoning error can trigger a cascade of failures, leading to dozens of unnecessary cycles and ultimately slowing down the system.
Huawei researchers identify an Efficiency Paradox: standard optimization techniques like INT8 quantization or aggressive context summarization, which typically benefit chatbots, can actually degrade agent performance. This degradation forces agents into expensive self-correction loops, negating any gains in raw throughput. To address this, the authors introduce AgentInfer, a unified framework that co-designs reasoning architectures with system-level scheduling and compression.
The AgentInfer Framework
AgentInfer decomposes the optimization problem into four synergistic modules designed to improve end-to-end efficiency while maintaining reasoning quality.
- AgentCollab (Hierarchical Reasoning): This module utilizes a dual-model setup. A smaller, faster model handles routine tasks, while a larger model is reserved for high-level planning. A "Progress Check" mechanism monitors the small model; if it becomes stuck or fails to make progress, the task is automatically escalated to the large model.
- AgentCompress (Semantic Compression): Unlike standard summarization, this module asynchronously distills the agent's memory. It prioritizes preserving internal "reasoning memory" while pruning redundant environmental observations, keeping the context window lean without introducing cognitive ambiguity.
- AgentSched (Cache-Aware Scheduling): Standard schedulers often evict the KV-cache of long-running agents to accommodate shorter requests. AgentSched uses a "shadow-price" controller to balance prioritizing short jobs with preserving the memory of long-running agents, avoiding costly re-computation.
- AgentSAM (Speculative Decoding): This module employs a Suffix Automaton (SAM) to predict future tokens based on current and historical session patterns. To maintain low Time-to-First-Token (TTFT), the SAM is constructed in a background thread.
Key Insights: The Quantization Trap
One of the most significant findings is the "Quantization Trap." While moving from FP16 to INT8 quantization increases raw throughput by 45%, it results in a 30% drop in task success rate. This loss in accuracy forces the agent into extensive retry loops, increasing the average recovery time by 1000%. Consequently, the total time-to-solution increases by 70%, making the "faster" quantized model significantly slower in practice.
Similarly, the researchers observed a "Scissor Effect" in context summarization. While aggressive summarization reduces tokens per step, it can lead to a 3.5x increase in the number of turns required to solve a problem. These insights highlight that for autonomous agents, reasoning stability is a critical efficiency feature.
Results and Evaluation
The AgentInfer framework was evaluated on complex benchmarks including BrowseComp-zh (web browsing) and DeepDiver (iterative reasoning) using the openPangu model family.
- End-to-End Speedup: AgentInfer achieved a 1.8x to 2.5x speedup in task completion time compared to native LLM agent baselines.
- Token Efficiency: The framework reduced ineffective token consumption—tokens spent on failed paths or retries—by over 50%.
- Accuracy and System Performance: AgentCollab recovered approximately 85% of the accuracy gap between small and large models. Additionally, AgentSched improved the average KV-cache hit rate from 63% to 72%, stabilizing performance during long-horizon tasks.
Conclusion
AgentInfer demonstrates that optimizing autonomous agents requires a shift from intra-step metrics to holistic, end-to-end system performance. By prioritizing reasoning stability and co-designing the inference stack, it is possible to achieve significant speedups and token savings without sacrificing the accuracy required for complex, multi-turn tasks. Future work may explore "Self-Evolution," where agents use distilled memories to fine-tune themselves, further reducing the need for model escalation.
Summary updated