TL;DR
HyFunc is a novel framework that drastically accelerates LLM-based function calling by eliminating computational redundancies through a hybrid-model cascade and dynamic templating. By using a large model to distill user intent into a single "soft token" and offloading the remaining generation to a smaller model with on-the-fly syntax injection, HyFunc achieves an inference latency of just 0.828 seconds while boosting a 0.6B model's accuracy to 80.1%.
What Problem Does This Paper Address?
Agentic AI systems rely heavily on Large Language Models (LLMs) to translate natural language user requests into executable function calls (APIs). However, deploying these systems in real-time applications is severely hindered by high inference latency.
The authors identify three primary sources of information redundancy in standard function-calling workflows:
- Context Processing Redundancy: Models are repeatedly forced to process massive libraries of irrelevant function descriptions for every single request.
- Full-Sequence Generation Redundancy: Using a massive, slow model to generate an entire structured sequence is computationally excessive, especially since the core intent is often decided in the very first token.
- Syntactic Generation Redundancy: A large portion of any function call consists of predictable boilerplate syntax (e.g., function names, parentheses, commas). Wasting LLM compute to generate these tokens one-by-one is highly inefficient.
Methods
HyFunc introduces a two-stage framework—Offline Preparation and Online Inference—built around a hybrid-model cascade that pairs a large model ($LM_L$) with a smaller, efficient model ($LM_S$).
The online inference pipeline consists of three core strategies:
- Soft Token Distillation & Function Retrieval: Instead of generating a full response, $LM_L$ executes a single forward pass on the user prompt to produce its first hidden state—a "soft token" that encapsulates the user's intent. A lightweight, dual-encoder Multi-Layer Perceptron (MLP) retriever uses this soft token to rapidly filter the massive function library down to a small set of relevant candidates.
- Guided Generation via Continuous Prompting: To bridge the semantic gap between the two models, a linear projector maps the high-dimensional soft token from $LM_L$'s embedding space into $LM_S$'s space. This projected token acts as a continuous prefix prompt, guiding $LM_S$ without requiring it to perform heavy semantic reasoning.
- Selective Token Fine-Tuning & Dynamic Templating: During training, $LM_S$ is optimized using a masked loss function (Selective SFT) that forces it to learn exclusively from tokens representing actual parameter values. At inference time, an extended vLLM engine alternates between directly injecting the fixed syntactical template and letting $LM_S$ autoregressively generate the parameter values. KV caching is utilized to prevent redundant attention computations during these mode switches.
Datasets
- Training / Offline Preparation: The authors utilized a dataset provided by xLAM containing 11,300 user queries. This dataset covers diverse function-calling scenarios, including multi-candidate tools and parallel function executions.
- Evaluation: HyFunc was evaluated using the Berkeley Function Call Leaderboard (BFCL) dataset (Out-of-Domain). This benchmark includes both synthetic test cases (Non-Live) and real-world scenarios (Live), split into four categories: Simple, Multiple, Parallel, and Parallel Multiple.
Key Insights and Ideas
- The First Token Holds the Intent: The authors validated the hypothesis that high-level semantic reasoning (deciding what tool to use) is heavily compressed into the very first generated token's hidden state.
- Decoupling Reason from Structure: By separating high-level intent understanding ($LM_L$) from low-level structured text generation ($LM_S$), you can achieve the accuracy of a large model with the speed of a small model.
- Syntax is Deterministic, Parameters are Not: Generating rigid syntax like
{"name": ..., "arguments": ...}token-by-token is a waste of an LLM's generative power. Treating syntax as a deterministic template that wraps around the LLM's parameter generation eliminates syntax errors entirely and slashes latency.
Key Results
- Unmatched Latency: HyFunc achieved an end-to-end inference latency of 0.828 seconds, outperforming all baseline models (including Qwen3-8B at 1.984s and Qwen2.5-0.5B at 0.927s).
- Massive Accuracy Gains for Small Models: When applying HyFunc to a 0.6B parameter model (Qwen3-0.6B), its overall accuracy skyrocketed from 62.2% to 80.1%. This allowed it to outperform much larger public models like xLAM-7B-r (76.6%) and compete closely with ToolACE-8B (81.0%).
- Token Reduction: The dynamic templating mechanism successfully reduced the number of generated output tokens by 29.54% compared to the base model.
Limitations and Future Work
- Retriever Sensitivity: HyFunc's performance is highly dependent on the accuracy of the lightweight MLP retriever. If the retriever fails to fetch the correct function, errors cascade into the generation phase.
- Single-Turn Limitation: The current implementation focuses exclusively on single-turn function-calling scenarios. The authors plan to extend HyFunc to multi-turn dialogue settings where active clarification strategies are required.
- Handling Unnecessary Calls: In scenarios where a function call is completely unnecessary, the dynamic template mechanism can force the model to generate meaningless arguments, introducing redundant decoding.
Summary updated