Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning

Qinhao Zhou, Zihan Zhang, Xiang Xiang, Ke Wang, Yuchuan Wu, Yongbin Li

Abstract: Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.

Introduction: The Agentic Performance Gap

While open-source Large Language Models (LLMs) like Llama and OpenChat have made significant strides in general language tasks, they often falter when deployed as autonomous agents. In complex environments requiring multi-step planning and tool usage, smaller models (7B and 13B parameters) frequently suffer from hallucinations, fail to adhere to strict output formats, or become trapped in repetitive loops.

Comparison of Agent Overall Scores between open-source LLMs and API-based commercial models, highlighting the performance gap.

As shown in the performance comparison above, there is a substantial gap between lightweight open-source models and commercial giants like GPT-4. This research addresses this disparity by investigating whether targeted fine-tuning and advanced reasoning architectures can bridge the gap without necessitating massive increases in model scale.

Methodology

The authors propose a two-pronged strategy: improving the model's foundational agentic behavior through Supervised Fine-Tuning (SFT) and enhancing its high-level problem-solving via a multi-branch reasoning framework.

1. Agent-Specific Supervised Fine-Tuning (SFT)

High-quality data for agentic interactions is notoriously scarce. To overcome this, the researchers employed GPT-4 to simulate complex three-way "roleplay" dialogues. In these simulations, GPT-4 acted as a question generator, an action-making agent, and an environmental feedback provider. This synthetic data was then mixed with general instruction-tuning data at an optimized ratio to fine-tune models using Low-Rank Adaptation (LoRA). This process effectively established a "performance floor," significantly reducing low-level errors like formatting failures.

2. Multi-Branch Reasoning and Task Decomposition

To elevate the "performance ceiling," the study introduced a non-linear reasoning strategy. Instead of following a single, linear path of thought (the standard ReAct pattern), the model utilizes:

  • Task Decomposition: A planning module breaks down complex goals into manageable sub-tasks.
  • Backtracking: If environmental feedback suggests a chosen path is suboptimal or failing, the agent can "backtrack" to a previous state and explore a different reasoning branch.
Comparison between linear reasoning (left) and the proposed multi-branch reasoning with backtracking (right).

Experimental Evaluation

The methodology was rigorously tested using AgentBench, focusing on five diverse environments:

  • ALFWorld: Simulated household tasks.
  • WebShop: E-commerce navigation and purchasing.
  • Mind2Web: General web browsing.
  • Operating System (OS): Solving problems via Bash commands.
  • Database (DB): Executing SQL queries on real databases.

Key Results and Insights

The combination of SFT and multi-branch reasoning yielded impressive gains. On the Operating System task, the tuned Llama2-7B model's score jumped from 3.8 to 11.54. In the Webshop environment, the OpenChat-13B model reached a score of 31, vastly outperforming the direct Input/Output baseline of 5 and the standard ReAct method.

Key takeaways include:

  • SFT is Essential for Reliability: Fine-tuning is critical for teaching smaller models how to interact with environments without hallucinating or breaking format.
  • Reasoning Strategy Drives Success: The ability to decompose tasks and backtrack from errors allows 7B and 13B models to solve problems previously reserved for much larger models.
  • Small Models are Viable: A well-tuned 7B model with a smart reasoning loop can rival specialized agent models, achieving an average score of 19.53 across benchmarks.

Conclusion and Future Work

This research demonstrates that model size is not the only path to agentic intelligence. By focusing on high-quality synthetic data and non-linear reasoning paths, low-parameter models can become highly capable agents. Future research will explore the scalability of these techniques to 70B+ models and methods to reduce the computational latency associated with exploring multiple reasoning branches.

Summary updated

Abstract

Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.
BibTeX
@inproceedings{zhou2024enhancing,
  author = {Qinhao Zhou and Zihan Zhang and Xiang Xiang and Ke Wang and Yuchuan Wu and Yongbin Li},
  title = {{Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=9F2NyQDKJL},
}
Figure