Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments

Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, Yu Su

Abstract

Large Language Models (LLMs) face significant challenges when acting as agents in massive environments, such as databases with millions of rows or knowledge bases with billions of facts. These environments cannot be "linearized" into a text string to fit within an LLM's limited context window. This paper introduces a Middleware framework that treats tools as an intermediary layer, allowing LLMs to actively navigate and query complex environments. By implementing specialized toolsets and a novel "decoupled generation" strategy, the authors achieve up to a 2.8x performance improvement over state-of-the-art baselines, significantly narrowing the gap between open-source and proprietary models.

Introduction: The Scaling Problem in Agentic Tasks

The goal of creating generalist agents capable of solving real-world tasks is often hindered by environmental complexity. When an LLM cannot perceive the entire environment, it tends to hallucinate or fail to ground its reasoning. Current methods typically attempt to maximize context window usage or rely on ungrounded planning—neither of which scales to industrial-sized data structures. The authors propose shifting the paradigm from "memorizing" the environment to "navigating" it through a middleware layer.

The Middleware Framework

The proposed framework shields the LLM from environmental complexity by providing a suite of specialized tools.

Comparison between traditional linearized descriptions (Memorization) and the proposed Middleware framework (Tool-based Navigation).

Specialized Toolsets and Error Feedback

The system utilizes 12 tools for databases (e.g., get_distinct_values()) and 7 for knowledge bases (e.g., get_relations()). These tools allow the agent to "peek" at the data before finalizing an answer. Crucially, the framework incorporates Error Feedback: if a tool is invoked incorrectly, the system provides a descriptive error message, enabling the LLM to self-correct in subsequent turns.

Decoupled Generation

To address the syntax bottlenecks often found in smaller models, the authors introduce Decoupled Generation. This reasoning scheme splits the process into two steps:

  1. Thought Generation: The LLM generates a natural language thought describing its intended action.
  2. Action Selection: A separate prompt asks the LLM to select a valid action from a constrained list of permissible tool calls that match the generated thought. This prevents the hallucination of invalid syntax or parameters.

Evaluation Benchmarks

The framework was evaluated on two primary fronts:

  • BIRD: A complex database benchmark where the authors identified a subset of 1,534 questions requiring deep content access.
  • KBQA-AGENT: A new benchmark of 500 "hard" questions requiring complex multi-relation reasoning and aggregation (e.g., counting or superlatives).

Key Results and Insights

The Middleware approach yielded substantial performance gains across both proprietary and open-source models.

Performance comparison across models on Knowledge Bases (F1) and Databases (EX), highlighting the benefits of decoupled generation.
  • Navigation over Memorization: The results confirm that LLMs do not need to see the entire dataset; they simply need the right "flashlight" to gather necessary information on demand.
  • Empowering Open Source: Decoupled generation proved particularly effective for open-source models. For instance, Mixtral-8x7B saw its F1 score on KB tasks jump from near-zero to 34.3, outperforming the standard GPT-3.5 baseline.
  • Closing the Oracle Gap: In database tasks, the gap between having "Oracle" knowledge (knowing the answer's location) and having no prior knowledge was reduced from 15.5% to just 4.0% using GPT-4 with middleware.

Conclusion and Future Directions

While the Middleware framework significantly enhances agentic performance, it currently relies on manually designed tools. Future research may explore "autonomous tool-making," where LLMs develop their own middleware for novel environments. Additionally, extending this approach to unstructured data and reducing the latency associated with multi-turn interactions remain key areas for development.

Summary updated

Abstract

The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist language agents capable of operating within complex real-world environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, this paper investigates the intriguing potential of tools to augment LLMs in handling such complexity. To this end, we design customized tools to aid in the proactive exploration within these massive environments. Such tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments -- knowledge bases (KBs) and databases -- we demonstrate the significant potential of augmenting language agents with tools in complex environments. Notably, equipped with these tools, GPT-4 achieves 2.8X the performance of the best baseline in tasks requiring access to database content and 2.2X in KB tasks. Our findings illuminate the path for advancing language agents in complex real-world applications.
BibTeX
@inproceedings{gu2024middleware,
  author = {Yu Gu and Yiheng Shu and Hao Yu and Xiao Liu and Yuxiao Dong and Jie Tang and Jayanth Srinivasa and Hugo Latapie and Yu Su},
  title = {{Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.436},
}
Figure