TL;DR
I-PTC (Interactive Programmatic Tool Calling) transforms how LLM agents use tools by providing them with a persistent, notebook-like execution environment. By allowing agents to maintain state, inspect intermediate results, and "learn" from API errors, I-PTC significantly improves success rates on complex tasks while reducing token consumption by up to 3x compared to standard baselines.
What Problem Does This Paper Address?
Current Large Language Model (LLM) agents typically interact with tools in one of two ways. Conventional tool calling is a "chatty" process: the model issues one call, sees the result in its prompt, and then decides on the next call. This is inefficient for complex tasks involving loops or data aggregation.
Programmatic Tool Calling (PTC) improved this by letting models write entire scripts to handle workflows. However, standard PTC is usually "stateless"—every time the model runs a piece of code, it starts in a fresh sandbox. If a script fails halfway through due to a minor API quirk, the agent often has to restart from scratch. Furthermore, serializing every intermediate data point back into the prompt quickly exhausts context windows. The authors argue that real-world problem-solving is inherently stateful, and our tool-use interfaces should reflect that.
Methods
The authors introduce I-PTC, a framework that treats tool use as an interactive, stateful session rather than a series of isolated events. It consists of three core mechanisms:
- Stateful Code Execution: Much like a Jupyter Notebook, I-PTC maintains a persistent Python namespace. Variables, imports, and function definitions created in one "cell" remain available in the next. This allows agents to cache expensive API results and recover from errors without losing progress.
- On-Demand Tool Inspection: Instead of cramming hundreds of API definitions into the initial system prompt (which is expensive and distracting), agents can "pull" documentation only when needed using a syntax similar to IPython (e.g.,
api.method??). - Self-Evolving APIs: When an agent encounters a recoverable error or discovers an undocumented API behavior, an "updating agent" generates a concise usage note. This correction is appended to the tool's documentation for future tasks, allowing the system to "learn" the environment's quirks over time.
Datasets
To test this framework, the authors introduced PTC-BENCH, a new benchmark featuring 240 production-style tasks across domains like calendar scheduling, travel planning, and organizational analysis. These tasks are designed to be "long-horizon," often requiring over 50 API calls to solve. The benchmark includes:
- Main Queries: Complex single-turn orchestration.
- Multi-turn Queries: Tasks requiring the reuse of information from a previous user follow-up.
- Error-Recovery Queries: Tasks where deterministic API failures are injected to test the agent's resilience.
The authors also evaluated I-PTC on existing benchmarks like MCP-Universe and MCP-Bench to demonstrate generalization.
Key Insights and Ideas
- State as a Token-Saver: One of the biggest "aha" moments is that keeping intermediate data in the execution environment's memory (as Python variables) is far more efficient than serializing it into the LLM's conversation context. This prevents "context pollution" and keeps the model focused on the high-level logic.
- Exploratory Programming for Agents: By allowing agents to "probe" data (e.g., printing the first two items of a list) before committing to a massive processing script, I-PTC enables a more human-like, iterative debugging process.
- API Documentation is Dynamic: The "Self-Evolving API" feature recognizes that human-written documentation is often incomplete. Treating the tool interface as a learnable state allows the agent collective to become more robust over time without retraining the underlying model.
Key Results
The results across ten different model families (including GPT, Gemini, and Claude) show a clear trend: I-PTC is both more accurate and more efficient.
- Performance Boost: On PTC-BENCH, GPT-5.4's pass rate jumped from a dismal 18.6% with traditional tool calling to 80.0% with I-PTC.
- Token Efficiency: I-PTC dramatically reduced "trajectory tokens" (the unique information the agent must process). For Claude Opus 4.7, trajectory tokens dropped from 30K to just 8K.
- Resilience: In error-recovery scenarios, I-PTC models were significantly more likely to succeed because they could inspect the error, fix the specific line of code, and continue, rather than getting stuck in a loop of repeated failures.
Limitations and Future Work
The authors acknowledge several hurdles for future research:
- Stale State: Just like in human-used notebooks, persistent state can lead to "hidden state" errors where a variable from a previous cell causes unexpected behavior in a later one.
- Model Capability Floor: Programmatic interfaces require strong reasoning. The authors found that models smaller than 4B parameters (like Qwen3.5-0.8B) actually perform worse with PTC than with simple tool calling, as they struggle to generate valid code.
- Documentation Integrity: The self-evolving API mechanism is currently lightweight; if the "updating agent" writes an incorrect correction, it could potentially mislead future agents. Future work is needed to validate these updates before they are persisted.
Summary updated