Abstract
The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist language agents capable of operating within complex real-world environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, this paper investigates the intriguing potential of tools to augment LLMs in handling such complexity. To this end, we design customized tools to aid in the proactive exploration within these massive environments. Such tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments—knowledge bases (KBs) and databases—we demonstrate the significant potential of augmenting language agents with tools. Notably, equipped with these tools, GPT-4 achieves 2.8X the performance of the best baseline in tasks requiring access to database content and 2.2X in KB tasks. Our findings illuminate the path for advancing language agents in complex real-world applications.
Introduction
As Large Language Models (LLMs) transition into autonomous agents, they must interact with massive real-world environments, such as databases containing millions of rows or knowledge bases (KBs) with billions of edges. The conventional approach to this problem is "linearization"—converting structured data into a text string to be included in the model's context window. However, this approach faces severe scalability limits, as context windows cannot accommodate terabyte-scale data. Furthermore, raw data dumps are an inefficient way for models to perceive complex environments.
This research argues that LLMs should function as "navigators" rather than "memorizers." By introducing a Middleware Layer, agents can proactively explore and query complex environments, fetching only the necessary information to solve a specific task.
The Middleware Framework
The proposed middleware layer consists of specialized tools tailored for specific data environments:
- Databases (12 tools): Includes navigational tools like
find_columns_containing_valuefor content exploration and functional tools likewhereandselectfor incremental SQL construction. - Knowledge Bases (7 tools): Includes tools for graph traversal (e.g.,
get_neighbors) and set operations (e.g.,intersection,count).
To enhance the reliability of tool usage, the authors implemented two key reasoning schemes:
- Error Feedback: Raw system errors (e.g., SQLite error messages) are returned to the LLM, enabling autonomous self-correction.
- Decoupled Generation: To prevent "hallucinated" tool calls, the process is split. The LLM first generates a reasoning "thought," and a subsequent prompt restricts its action selection to a list of valid, permissible candidates.
Evaluation and Results
The framework was evaluated on two challenging benchmarks: BIRD (for large-scale databases) and KBQA-AGENT (a new benchmark for complex, multi-hop knowledge base queries requiring at least three relations or complex aggregations).
Key Findings
- Significant Performance Gains: On content-heavy SQL tasks, GPT-4 with Middleware achieved a 38.3% execution accuracy, a 2.8x improvement over the standard "API Docs" baseline (13.8%). On KBQA-AGENT, it reached an F1 score of 59.3, more than doubling the previous state-of-the-art.
- Empowering Open-Source Models: The "Decoupled Generation" scheme significantly boosted the reliability of smaller models. For instance, Mixtral-8x7B nearly matched the performance of GPT-3.5-turbo when using this framework.
- Active Exploration Efficiency: The results confirm that providing LLMs with "sensory organs" (tools) to fetch relevant data bits is far more effective than passive context ingestion.
Conclusion
The middleware approach effectively shields LLMs from environmental noise and complexity. While current tools are manually designed, future work may explore autonomous tool-making by LLMs. Despite increased latency and token costs associated with multi-turn interactions, the dramatic performance improvements underscore the necessity of tool-based navigation for language agents in complex real-world applications.
Summary updated