TL;DR
This survey provides the first comprehensive taxonomy of optimization techniques for Large Language Model (LLM)-based agents, dividing them into parameter-driven (fine-tuning and reinforcement learning) and parameter-free (prompting and retrieval) methods. It maps out how researchers are moving beyond vanilla next-token prediction models to build autonomous systems capable of long-horizon planning, tool use, and robust environmental interaction.
What Problem Does This Paper Address?
While vanilla LLMs excel at general language understanding, they are not inherently designed for autonomous decision-making or long-term execution. Their primary training objective—next-token prediction—frequently falls short in complex agentic workflows where an agent must plan strategically, select external tools autonomously, and adapt to dynamic environments.
When deployed as agents, open-source models often suffer from performance degradation over extended interactions, limited memory retention, and a lack of adaptability to novel environments. While numerous individual strategies (like supervised fine-tuning or prompt engineering) have been proposed to address these gaps, the community has lacked a holistic, systematic review that treats agent optimization as its own distinct research paradigm. This paper fills that gap.
Methods
The authors organize the vast landscape of agent optimization into a clear, two-pronged taxonomy:
1. Parameter-driven Optimization
This paradigm focuses on adjusting the actual weights of the LLM to bake in agentic capabilities. It is further broken down into:
- Conventional Fine-Tuning: Utilizing trajectory datasets formatted for Supervised Fine-Tuning (SFT) or Parameter-Efficient Fine-Tuning (PEFT, like LoRA). This process relies heavily on high-quality trajectory data construction, evaluation, and filtering.
- Reinforcement Learning (RL)-based Optimization: Moving away from static datasets to let agents learn from environmental interactions. This includes traditional reward-function-based methods (e.g., PPO, Actor-Critic) and offline preference alignment methods (e.g., Direct Preference Optimization - DPO) that rank successful vs. failed trajectories.
- Hybrid Fine-Tuning: An emerging approach that combines SFT (for a stable behavioral warm-up) with subsequent RL or DPO iterations to refine the agent's policy dynamically.
The complete workflow for fine-tuning-based agent optimization is illustrated below:
2. Parameter-free Optimization
This paradigm enhances agent behavior at inference time without modifying model weights, making it highly efficient for resource-constrained setups. It includes:
- Experience-based: Leveraging historical trajectories stored in memory modules (e.g., knowledge graphs).
- Feedback-based: Utilizing self-reflection loops, external critics, or meta-prompt optimization to iteratively correct agent actions.
- Tool-based & Retrieval-based: Optimizing how agents invoke external APIs and integrate dynamic knowledge retrieval (such as RAG or deep web exploration workflows).
- Multi-Agent Collaboration: Distributing roles across multiple specialized sub-agents to solve complex tasks collectively.
An overview of how these sections are structured throughout the paper can be seen in the organization chart:
Datasets
The survey compiles a comprehensive list of resources used for both training and evaluating LLM agents:
- Tuning Datasets: Trajectory-focused corpora such as AgentInstruct, AgentBank, Agent-FLAN, AgentOhana, and ToRA-CORPUS. These datasets standardize multi-turn formats and cover heterogeneous action spaces to support scalable behavioral cloning.
- Evaluation Benchmarks: The paper categorizes benchmarks by domain, including Math (GSM8K, MATH), Code (SWE-bench, HumanEval), Tool-use (T-Eval, ToolEval), Web Navigation (WebArena, Mind2Web), and Environment Interaction (ScienceWorld, ALFWorld).
- Multi-Task Suites: Holistic evaluation frameworks like AgentBench, AgentEval, and GAIA that test an agent's generalized ability to reason, navigate the web, and use tools simultaneously.
Key Insights and Ideas
- The "Data Mixing" Mandate: Fine-tuning an LLM exclusively on agent trajectory data severely degrades its general language and reasoning capabilities. To maintain foundational skills, researchers must mix general instruction datasets with task-specific agent trajectories during training.
- Failing Forward with Low-Quality Data: Failed or suboptimal trajectories shouldn't just be thrown away. Modern optimization strategies use contrastive learning or preference alignment (like DPO) to show agents explicit pairs of "what to do" versus "what not to do," significantly reducing hallucinations and invalid tool calls.
- The Rise of Offline Preference Alignment: While online RL (like PPO) is highly effective for policy optimization, it is computationally expensive and sensitive to reward design. Offline preference alignment (DPO, KTO) is rapidly gaining traction because it simplifies training by treating preference pairs as a direct proxy for rewards without requiring online environment rollouts.
Key Results
As a survey paper, the work focuses on synthesizing existing literature rather than introducing a new empirical model. Key analytical takeaways include:
- Data Strategy Trade-offs: The authors provide a structured comparison of data generation strategies, noting that while expert-annotated data ensures the highest accuracy, self-exploration and strong LLM generation are essential for scaling data volume at a lower cost.
- Data Filtering Paradigms: They map out the pros and cons of environment-based, rule-based, and model-based data filtering, showing a clear trend toward hybrid evaluation strategies to ensure data robustness.
- Cross-Domain Impact: The survey highlights how optimized agents are achieving state-of-the-art automation in specialized fields:
- Healthcare: Simulating clinical environments (e.g., Agent Hospital).
- Science: Automating gene-editing design and protein discovery (e.g., CRISPR-GPT).
- Embodied AI: Enabling continuous, open-ended skill acquisition in robotics (e.g., Voyager).
Limitations and Future Work
The authors identify several critical bottlenecks that define the next frontier of agent research:
- Algorithmic Multi-Turn Limitations: Current preference alignment methods like DPO are primarily suited for single-step optimization. Extending these algorithms to effectively handle multi-turn, long-horizon interactive tasks remains a major challenge.
- Lack of Standardized Metrics: Existing benchmarks mostly evaluate final task completion rather than the extent of optimization or stepwise progress, making fair cross-task comparisons difficult.
- Cost and Latency Constraints: Extended reasoning chains and multi-turn interactions explode token consumption and inference depth, limiting the practicality of deploying advanced agents in production environments.
- Multi-Agent Joint Optimization: Most current optimization frameworks target a single agent. Joint parameter tuning, reward-sharing mechanisms, and coordination structures for Multi-Agent Systems (MAS) are still largely underexplored.