Enhancing the General Agent Capabilities of Low-Paramter LLMs through Tuning and Multi-Branch Reasoning

Qinhao Zhou, Zihan Zhang, Xiang Xiang, Ke Wang, Yuchuan Wu, Yongbin Li

Abstract

Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.

Introduction

While Large Language Models (LLMs) like GPT-4 excel as autonomous agents—planning tasks, using tools, and navigating environments—smaller open-source models (7B–13B parameters) typically fail in these roles. These "low-parameter" models often suffer from three main issues: formatting errors, hallucinations, and reasoning bottlenecks. As shown in Figure 1, there is a significant performance gap between commercial API-based models and existing open-source alternatives in agentic tasks.

Comparison of agent performance across various LLMs, highlighting the gap between commercial and open-source models.

The goal of this research is to bridge this performance gap without needing massive increases in model size, making capable AI agents more accessible and deployable on consumer-grade hardware.

Methodology

The proposed framework addresses the limitations of small LLMs through two primary strategies: data-driven fine-tuning and advanced reasoning strategies.

Supervised Fine-Tuning (SFT) with Synthetic Data

High-quality agent interaction data is scarce. To overcome this, the researchers used GPT-4 to simulate agentic behavior within a multi-role framework. In this setup, GPT-4 plays three roles: a Question Generator to set goals, an Action Maker to decide steps, and an Environmental Agent to provide feedback.

Models like Llama-2 were fine-tuned using a mixture of this synthetic agent data and general instruction data. This "mixing" is crucial; training exclusively on agent data can cause the model to lose its general reasoning abilities. The researchers identified a specific "mix ratio" ($\lambda$) required to balance agentic performance with general intelligence.

Multi-Branch Reasoning and Task Decomposition

Standard agents often rely on linear reasoning paths like "ReAct" (Reason + Act). However, if an agent makes a single mistake in a linear chain, the entire task typically fails. To improve robustness, the researchers proposed:

  1. Task Decomposition: A separate planning LLM breaks a complex goal into manageable sub-goals.
  2. Multi-Branch Reasoning with Backtracking: Instead of a single path, the model explores multiple reasoning branches. If it reaches a suboptimal state or fails a sub-goal, it is prompted to recognize the failure, return to a previous node in its reasoning tree, and attempt a different path.
Comparison between standard linear reasoning and the proposed multi-branch reasoning with backtracking.

Evaluation and Results

The models were evaluated using AgentBench, testing across five domains: ALFWorld (household tasks), WebShop (e-commerce), Mind2Web (web browsing), Operating System (Bash commands), and Database (SQL operations).

The improvements were dramatic. For the Llama2-7b model, the average score across all tasks jumped from 2.43 (base model) to 18.33 after tuning. While still trailing GPT-4 (48.50), the tuned 7B model outperformed base 13B models and approached the performance of much larger open-source alternatives.

Key findings include:

  • Formatting Fix: SFT significantly reduces formatting errors, which are a primary cause of failure in small models.
  • Reasoning Efficiency: On the WebShop task, the multi-branch method achieved a score of 27.00, significantly higher than standard ReAct (13.35) or CoT (4.0) on the same 7B model.
  • Decomposition Benefits: Breaking tasks down helped the 13B model reach a score of 15.26 on the average benchmark, a notable lead over standard linear methods.

Conclusion

This research demonstrates that through high-quality synthetic data fine-tuning and the implementation of multi-branch reasoning with backtracking, low-parameter LLMs can achieve significantly enhanced agent capabilities. These techniques effectively reduce hallucinations and formatting errors while providing a self-correction mechanism that was previously the domain of much larger models. Future work will explore the scalability of these methods to larger models and the optimization of data mixing ratios.

Summary updated

Abstract

Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.
BibTeX
@inproceedings{zhou2024enhancing,
  author = {Qinhao Zhou and Zihan Zhang and Xiang Xiang and Ke Wang and Yuchuan Wu and Yongbin Li},
  title = {{Enhancing the General Agent Capabilities of Low-Paramter LLMs through Tuning and Multi-Branch Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=9F2NyQDKJL},
}
Figure