TL;DR
This paper introduces a new benchmark to evaluate the robustness of Large Language Model (LLM) agents that use function calling (FC). They found that even state-of-the-art LLMs struggle when faced with natural variations in user queries or when the available tools become more complex, highlighting critical weaknesses in current evaluation methods and real-world agent deployments.
What Problem Does This Paper Address?
LLMs are increasingly being used as autonomous agents that can interact with the real world by calling external tools (functions). While much research has focused on improving the accuracy of these function calls, little attention has been paid to their robustness. Robustness, in this context, means how well an agent can perform its task when faced with slight, natural changes in the input or its environment. The authors identify two key areas where robustness is crucial but understudied:
- Naturalistic Query Variations: Users don't always phrase their requests perfectly. They might use synonyms, rephrase sentences, or introduce minor variations. A robust agent should still understand the intent and call the correct function.
- Toolkit Expansion: Real-world systems might have thousands of tools. Agents typically use a "shortlisting" mechanism to select a relevant subset. How does an agent perform when this subset includes semantically similar tools, potentially leading to confusion?
Current benchmarks often use simplified toolkits and don't adequately test these real-world challenges, leading to an overestimation of agent capabilities.
Methods
The researchers tackle this problem by creating a new benchmark and evaluating several top LLMs on it. Their approach involves two main components:
- User Query Perturbations: They generate meaning-preserving paraphrases of existing user queries. The key here is that the paraphrased query must still map to the exact same function and parameter values as the original. This is achieved using LLMs like Llama3.1-70B with carefully crafted prompts.
- Toolkit Expansion: They simulate a more realistic scenario where the agent's toolkit is expanded with semantically related tools. This is done by:
- Generating related but distinct user requests.
- Using another LLM (CodeLlama-13B) to generate function definitions for these new requests, ensuring they match the style and format of existing tools.
- Filtering out tools that are too similar to existing ones to ensure the expansion adds meaningful complexity.
They then evaluate LLM agents on this expanded benchmark using the Abstract Syntax Tree (AST) matching methodology, which checks if the generated function call structure is correct.
Datasets
The primary dataset used is a subset of the Berkeley Function Calling Leaderboard (BFCL). The authors then augment this dataset by creating two new variants:
- Rephrased Queries: Original BFCL queries are paraphrased.
- Expanded Toolkits: The original BFCL toolkits are expanded with semantically related tools.
The paper also makes their benchmark data publicly available at https://huggingface.co/datasets/ibm-research/BFCL-FC-robustness.
Key Insights and Ideas
- Evaluation Matters: The authors found that a significant portion of errors (70-90%) in their experiments stemmed from the evaluation approach itself, particularly in how it handles argument matching, rather than the LLM agent's inherent inability. This suggests current FC evaluation might be too strict on exact matches for things like named entities.
- Robustness is a Real Problem: Even with minor, meaning-preserving query variations, performance drops significantly. This indicates that LLM agents are more brittle than often assumed.
- Toolkit Complexity is a Challenge: Expanding the toolkit with semantically related tools consistently degraded performance across all tested LLMs. This suggests that while LLMs can generate function calls, selecting the correct one from a larger, more nuanced set is a substantial hurdle.
- Failure Modes are Diverse: When agents fail with expanded toolkits, the errors aren't just one type. They include selecting the wrong function, generating the wrong number of functions, incorrect parameter assignment, and even hallucinating parameters.
Key Results
The paper reports AST performance scores for various LLMs under different conditions. While specific numbers vary by model, the general trend is clear:
- Original Toolkit + Rephrased Query: Performance drops noticeably compared to the original query with the original toolkit. For example, Llama3.1-70B drops from 0.965 to 0.825 (a 15% drop).
- Original Toolkit + Original Query: This serves as the baseline.
- Expanded Toolkit + Original Query: Performance also drops compared to the baseline, indicating challenges with a more complex tool environment. For instance, Llama3.1-70B drops to 0.925 (a 4% drop).
- Error Breakdown: In the expanded toolkit scenario, failures are distributed across wrong function selection and wrong parameter assignment, with no single error type dominating.
Limitations and Future Work
The authors acknowledge several limitations:
- Single Dataset: The evaluation was performed on a single dataset, and more diverse datasets are needed for broader analysis.
- Offline Toolkit Expansion: The process of generating expanded toolkits is time-consuming and currently done offline. They are exploring ways to streamline this.
Future work includes:
- Testing robustness on additional and diverse datasets.
- Investigating agent behavior when the toolkit is expanded with non-relevant tools to compare performance against the current setting.
- Exploring semantic similarity-based evaluation methods to address the limitations of exact matching in argument validation.
Summary updated