Abstract
Digital tool-based agents that invoke external Application Programming Interfaces (APIs) often rely on documentation to understand API functionality. However, such documentation is frequently missing, outdated, privatized, or inconsistent—hindering the development of reliable, general-purpose agents. In this work, we propose learning API functionality directly from demonstrations as a new paradigm applicable in scenarios without documentation. Using existing API benchmarks, we collect demonstrations from both expert API-based agents and from self-exploration. To understand what information demonstrations must convey for successful task completion, we extensively study how the number of demonstrations and the use of LLM-generated summaries and evaluations affect the task success rate of the API-based agent. Our experiments across 3 datasets and 5 models show that learning functionality from demonstrations remains a non-trivial challenge, even for state-of-the-art LLMs. We find that providing explicit function calls and natural language critiques significantly improves the agent's task success rate due to more accurate parameter filling. We analyze failure modes, identify sources of error, and highlight key open challenges for future work in documentation-free, self-improving, API-based agents.
Introduction
Most current LLM-based agents rely heavily on high-quality API documentation to understand which tools to pick and how to format their inputs. However, in real-world enterprise environments or rapidly evolving software ecosystems, documentation is frequently missing, privatized, or inaccurate. This paper addresses the "documentation-free" scenario by formalizing the problem of learning API functionality from scratch using only demonstrations of tool calls. The goal is to determine if an agent can infer the underlying schema and logic of an API from a few examples well enough to complete new, complex tasks.
Methods
The researchers explored several strategies for processing expert demonstrations and learning from experience:
- Direct Expert Demonstrations (DxD): Feeding raw examples of task-call-result sequences directly into the LLM's prompt.
- Generated Documentation (GD): Using a "generator" LLM to synthesize a manual for the agent based on observed demonstrations.
- Generated Documents with Example Calls (GDEC): A hybrid approach providing both the generated manual and raw example calls.
- Self-Exploration & Critiques: The agent attempts training tasks and receives natural language critiques from an "LLM Judge." These critiques (e.g., "You forgot the ID prefix") are used to update the agent's internal guidelines.
Experimental Setup
The study utilized three diverse API benchmarks to evaluate performance:
- WorkBench: Focused on workplace tasks like managing emails, calendars, and analytics.
- τ-Bench (retail): A retail-oriented benchmark involving multi-step queries like canceling orders or looking up user IDs.
- CRMArena: A benchmark designed for professional Customer Relationship Management tasks in realistic environments.
Key Results and Insights
Learning from demonstrations remains a significant challenge. While agents achieved success rates (SR) in the 40-60% range on WorkBench and CRMArena using direct demonstrations, they lagged behind "Oracle" baselines with access to ground-truth documentation.
The Parameter Filling Bottleneck
A primary hurdle is formatting arguments correctly. For instance, in τ-Bench, success rates jumped from 10% to 49% by simply hardcoding a single formatting rule (a # prefix for order IDs) that agents failed to infer from demonstrations alone. This highlights how a single missing character can cause a massive drop in success rates.
The Role of Critiques and Data Volume
- Power of Critiques: Natural language feedback on failures significantly improved self-correction. Using "Attached Guidelines" (AG)—summarized lessons from self-exploration—showed a 12% improvement over standard generated documentation in WorkBench.
- Diminishing Returns of Data: Surprisingly, more demonstrations sometimes decreased performance. Generator LLMs occasionally hallucinated that optional parameters were mandatory, leading to runtime errors in new contexts.
- Model Resilience: In some tests, the open-source model Qwen3 actually outperformed state-of-the-art OpenAI models on the WorkBench dataset, suggesting that reasoning capabilities during action generation are as critical as the quality of the learned documentation.
Conclusion and Future Work
The authors identify several key challenges for future development. Currently, systems generate documentation for each function independently; future work could explore "group learning" to share insights across a whole toolset. Additionally, incorporating human-in-the-loop feedback could be a powerful path for training agents on proprietary corporate tasks where expert demonstrations are noisy or suboptimal.
Summary updated