Abstract
Agents backed by large language models (LLMs) often rely on external tools drawn from marketplaces where multiple providers offer functionally equivalent options. This raises a critical point concerning fairness: if selection is systematically biased, it can degrade user experience and distort competition by privileging some providers over others. We introduce a benchmark of diverse tool categories, each containing multiple functionally equivalent tools, to evaluate tool-selection bias. Using this benchmark, we test seven models and show that unfairness exists with models either fixating on a single provider or disproportionately preferring earlier-listed tools in context. To investigate the origins of this bias, we conduct controlled experiments examining tool features, metadata (name, description, parameters), and pre-training exposure. We find that: (1) semantic alignment between queries and metadata is the strongest predictor of choice; (2) perturbing descriptions significantly shifts selections; and (3) repeated pre-training exposure to a single endpoint amplifies bias. Finally, we propose a lightweight mitigation that first filters the candidate tools to a relevant subset and then samples uniformly, reducing bias while preserving good task coverage. Our findings highlight tool-selection bias as a key obstacle for the fair deployment of tool-augmented LLMs.
The Problem of Tool-Selection Bias
As LLMs evolve into autonomous agents, they increasingly rely on external "tools" (APIs) to perform tasks like checking the weather, booking flights, or searching databases. In a healthy digital marketplace, multiple providers often offer functionally identical services. However, if an LLM agent is systematically biased—preferring "WeatherAPI_A" over "WeatherAPI_B" simply because of its name, its description, or its position in a prompt—it creates a "winner-take-all" dynamic that stifles competition and potentially degrades user experience.
While social and cognitive biases in LLMs have been studied extensively, tool-selection bias remains a relatively unexplored frontier. This research seeks to uncover whether these models are "fair" shoppers in the API marketplace and identifies the triggers behind their preferences.
Methodology: The BiasBusters Framework
The researchers developed a framework called BiasBusters to audit and fix these selection imbalances. Their approach involves several key components:
- Quantification: Using Total Variation (TV) distance, the study measures how far a model's selection distribution deviates from a "fair" uniform distribution. This tracks both API-level bias (preferring a specific provider) and positional bias (preferring the first tool listed in the prompt).
- Diagnostic Probing: To find the root causes, the team performed attribute-level regressions and metadata perturbations. By systematically scrambling tool names, descriptions, and parameters, they identified which changes caused the model to switch its preferred tool.
- Biased Continued Pre-training (CPT): A model was deliberately trained on a corpus saturated with a specific API to determine if "fame" in the training data translates to favoritism during inference.
- Mitigation: A "decoupling" strategy was proposed. Instead of asking one model to both identify and select a tool, a lightweight "selector" model identifies a subset of valid tools, and the system then samples uniformly from that subset.
Datasets and Evaluation
The study utilizes a newly curated benchmark built on the ToolLLM pipeline, consisting of 10 clusters (each containing 5 functionally equivalent APIs) and 1,000 provider-agnostic user queries. Seven frontier models were tested, including ChatGPT (3.5 and 4o-mini), Claude 3.5 Sonnet, Gemini 1.5 Flash, DeepSeek R1, ToolLLaMA, and Qwen2.5.
Key Insights
- Semantic Alignment is the Primary Driver: The strongest predictor of selection is how closely the wording of a tool's description matches the user's query. Tools using "buzzwords" that mirror common user prompts gain an unfair advantage.
- The Bias Trade-off: Models tend to fall into two regimes: they either have a strong "intrinsic" preference for a specific API regardless of its position, or a strong "positional" preference, defaulting to the first tool in the list when no specific API stands out.
- Training Exposure Matters: The "Biased CPT" experiment proved that repeated exposure to an API's metadata during training significantly boosts its selection rate, suggesting a built-in "incumbency advantage" for popular APIs found in web-scale training data.
- Names are Secondary: Interestingly, scrambling tool names had a much smaller impact than scrambling descriptions. Models appear to evaluate the fine print of a tool's capabilities more than its brand name.
Results and Mitigation Success
All seven models exhibited substantial bias. On average, roughly 30–40% of the selection probability mass would need to be redistributed to achieve true fairness. ChatGPT 4o-mini was identified as the most biased model, while Qwen2.5 (72B) was the most balanced.
The proposed "filter-then-sample" method proved highly effective. For ChatGPT 4o-mini, the combined bias metric plummeted from 0.380 to 0.094 while maintaining a near-perfect precision of ~99% in tool relevance. This suggests that fairness can be achieved without sacrificing the functional accuracy of the agent.
Conclusion
Tool-selection bias represents a significant hurdle for the fair deployment of tool-augmented LLMs. By identifying the drivers of this bias—particularly semantic alignment and training exposure—and providing a lightweight mitigation strategy, this work lays the groundwork for more equitable AI agent ecosystems. Future work will focus on scaling the benchmark to more diverse tool categories and investigating how these biases evolve as models are trained on increasingly agentic, tool-heavy datasets.
Summary updated