Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments

Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, Yu Su

TL;DR

Large Language Models (LLMs) struggle to act as agents in massive environments like databases or knowledge bases because the data far exceeds their context window. This paper introduces "middleware" tools—a specialized set of functions that allow LLMs to actively explore and query complex environments, resulting in a 2.8x performance boost for GPT-4 on database tasks and a 2.2x boost on knowledge base reasoning.

What Problem Does This Paper Address?

The current trend in AI is moving from LLMs as text processors to LLMs as agents that can navigate the real world. However, real-world environments—such as a database with millions of rows or a knowledge base (KB) with billions of edges—are "complex."

The standard approach to grounding an LLM in an environment is linearization: converting the environment into a long string of text and stuffing it into the model's prompt. This fails for two reasons:

  1. Scalability: You simply cannot fit a terabyte-scale database into a context window.
  2. Perception: Raw text descriptions often don't capture the structural relationships of the data effectively.

The authors argue that for an LLM to be a successful agent, it shouldn't just "memorize" the environment via the prompt; it needs to "navigate" it using tools.

Figure 1: (left) When an LLM engages with a complex environment, it can develop an understanding by fitting the environment’s description (i.e., linearized tokens) into its short-term memory (i.e., the LLM’s input win- dow). However, this method encounters drastic scalabil- ity issues as the complexity of the environment grows. (right) Another option is to furnish the LLM with a set of tools that assist it in actively engaging with the envi- ronment and acquiring the necessary information.

Methods

The researchers propose a framework called MIDDLEWARE, which equips LLMs with a suite of environment-specific tools. These tools act as an abstraction layer, shielding the LLM from raw complexity while allowing it to gather information on demand.

1. The Toolsets

  • Databases (12 tools): Includes navigational tools (e.g., find_columns_containing_value()) to help the agent find where specific data lives, and functional tools (e.g., where(), select()) to help build SQL queries step-by-step.
  • Knowledge Bases (7 tools): Includes tools to explore relations (e.g., get_relations()) and perform operations like intersection() or count() on sets of entities.

2. Reasoning Schemes

To make tool use more reliable, the authors introduced two strategies:

  • Error Feedback: If a tool call fails (e.g., a malformed SQL query), the system feeds the specific error message back to the LLM, allowing it to self-correct.
  • Decoupled Generation: Instead of the LLM generating a thought and an action in one go, the process is split. The model first generates a "thought," and then a separate prompt asks it to choose the best "action" from a constrained list of valid candidates. This significantly helps weaker, open-source models stay within the bounds of valid syntax.

Datasets

The authors evaluated their framework on two challenging fronts:

  • BIRD (Big Bench for Large-scale Database Grounded Text-to-SQLs): A recent, complex database benchmark. The authors manually labeled a subset of 496 questions that strictly require accessing database content (not just the schema) to solve.
  • KBQA-AGENT: A new benchmark curated by the authors. It pulls 500 complex questions from existing datasets (like GrailQA) but filters for tasks requiring at least three relations or complex aggregations, making it much harder than standard "one-hop" benchmarks.

Key Insights and Ideas

  • Active Exploration > Passive Context: The most striking finding is that providing more raw data in the prompt (e.g., sampling 20 rows of a table) can actually hurt performance by cluttering the context. In contrast, giving the agent the tools to query that data leads to much higher accuracy.
  • Redundancy Breeds Robustness: In ablation studies, the authors found that removing a single tool often didn't crash the system. The agents were clever enough to use other tools to reach the same information, suggesting that a rich, redundant toolset makes for a more resilient agent.
  • Bridging the Open-Source Gap: While GPT-4 is naturally good at tool use, smaller models like Mistral-7B struggle. However, the "Decoupled Generation" scheme allowed these smaller models to reach performance levels comparable to GPT-3.5 baselines by preventing hallucinations of invalid tool calls.
Figure 3: The open-source LLMs still largely lag behind GPT-3.5-turbo and GPT-4 in both environments.

Key Results

The results demonstrate that the middleware approach is essential for tasks requiring deep environmental engagement:

  • Database Tasks: On questions requiring database content access, GPT-4 with MIDDLEWARE achieved 38.3% Execution Accuracy (EX), compared to just 13.8% for the best baseline (a 2.8x improvement).
  • Knowledge Base Tasks: On the KBQA-AGENT benchmark, GPT-4 with decoupled generation reached an F1 score of 59.3%, dwarfing the 27.1% achieved by Pangu, the previous state-of-the-art.
  • Self-Correction: Error feedback was highly effective for stronger models; GPT-4 was often able to fix its own SQL syntax errors when prompted with the database's error logs.
Table 1: Detailed results on the BIRD database benchmark.

Limitations and Future Work

  • Manual Engineering: The tools used in this study were manually designed by experts. The authors suggest that future work should explore autonomous tool-making, where the agent creates its own middleware as it learns the environment.
  • Interface Requirements: This approach relies on the environment having a structured query interface (like SQL or SPARQL). Applying this to unstructured environments (like a physical robot navigating a room) remains a challenge.
  • Efficiency: Tool-based navigation is slower and uses more tokens than single-shot prompting because it requires multiple rounds of interaction. Optimizing the "cost-per-task" is a necessary next step for production systems.

Summary updated

Abstract

The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist agents capable of operating within complex environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, we seek to investigate the intriguing potential of tools to augment LLMs in handling such complexity by introducing a novel class of tools, termed middleware, to aid in the proactive exploration within these massive environments. Such specialized tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments -- knowledge bases (KBs) and databases -- we demonstrate the significant potential of augmenting language agents with tools in complex environments. Notably, equipped with the middleware, GPT-4 achieves 2.8X the performance of the best baseline in tasks requiring access to database content and 2.2X in KB tasks. Our findings illuminate the path for advancing language agents in real-world applications.
BibTeX
@inproceedings{gu2024middleware,
  author = {Yu Gu and Yiheng Shu and Hao Yu and Xiao Liu and Yuxiao Dong and Jie Tang and Jayanth Srinivasa and Hugo Latapie and Yu Su},
  title = {{Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.436},
}
Figure