TL;DR
This paper introduces a data-driven pipeline designed to enhance the function-calling capabilities of Large Language Models (LLMs) within live financial question-answering (QA) systems. By automatically detecting dataset "blind spots" using information entropy and employing a flexible two-step training process (SFT + RL), the proposed framework significantly boosts tool-use accuracy and reliability. The pipeline has been successfully deployed in production on Tencent's YuanBao platform, one of China's largest chat applications.
What Problem Does This Paper Address?
Deploying LLMs for real-world financial QA is notoriously difficult. Financial data changes rapidly, requiring models to interface with highly customized, private APIs to fetch real-time information. Generic LLMs struggle with these specialized tools because they lack domain-specific training data.
Furthermore, production environments introduce massive query diversity. Users rarely format queries perfectly; they use company nicknames (e.g., calling Tencent "Goose Factory" or using its stock ticker "700") and input highly varied parameters. Because real-world user queries typically follow a power-law distribution, standard training datasets suffer from parameter distribution collapse—meaning a few common parameter values dominate the data, leaving "blind spots" where the model fails to generalize to rarer, long-tail inputs.
Methods
To overcome these challenges, the authors developed an automated, data-driven pipeline consisting of three core phases:
- Data Collection & Construction: The system continuously captures live user queries. To ensure data quality without overwhelming human annotators, queries are validated via embedding models and passed to a powerful teacher LLM for few-shot tool-call generation. Human experts are called in only to resolve inconsistencies between the production model and the teacher model.
- Automated Data Augmentation (AugFC): To eliminate parameter blind spots, the authors introduce AugFC. This method uses semantic clustering (K-Means) to group queries and calculates both global and local information entropy for parameter values. If a parameter's local entropy within a cluster is significantly lower than its global entropy, it is flagged as a "blind spot" (indicating value collapse). The system then prompts an LLM to generate distribution-aware counterfactual queries to fill these specific gaps.
- Two-Step Model Training:
- Supervised Fine-Tuning (SFT) with Prompt Isolation: The model is trained on a mix of reasoning-heavy samples (with Chain-of-Thought planning) and direct-calling samples. By isolating these styles into two distinct system prompts, developers can toggle off the reasoning tokens during low-latency inference modes to save computational overhead.
- Reinforcement Learning (RL): The model is optimized using Group Relative Policy Optimization (GRPO). It relies on a rule-based reward function that scores the output based on format compliance, correct tool selection, parameter key retrieval, and exact parameter value matching.
Datasets
The pipeline was evaluated across both offline benchmarks and live production data:
- Offline Benchmarks: API-Bank (Levels 1 and 2), Tool-Alpaca, Seal-Tools, Nexus Raven Evaluation, and a sampled subset of xLAM-small.
- Online Production Dataset: A proprietary dataset derived from live interactions on Tencent's YuanBao platform, evaluated using 350 end-to-end query-answer pairs comprising nearly 2,000 tool-call instances.
Key Insights and Ideas
- The Power-Law Trap: Real user interactions naturally cluster around a tiny fraction of possible parameter values. Relying solely on raw user logs for fine-tuning inadvertently trains models that fail on rare but critical API arguments.
- Entropy as a Blind-Spot Radar: Instead of guessing where a dataset lacks diversity, calculating the ratio of local-to-global information entropy mathematically pinpoints exactly which parameters have collapsed within specific semantic contexts.
- Prompt Isolation for Latency Control: Chain-of-Thought (CoT) reasoning improves tool-calling accuracy but introduces severe token latency. Training the model to handle dual system prompts (one with a
<plan>tag and one without) allows production systems to dynamically balance accuracy and speed depending on the query type.
Key Results
Offline Performance
When applied to base models like Qwen2.5 (ranging from 1.5B to 32B parameters), the combined dataset (src+aug) consistently outperformed models trained purely on seed data. The two-step training method also beat out established function-calling models like xLAM and Hammer across the six offline benchmarks.
Online Production Gains
In live deployment on the YuanBao platform, the data-driven pipeline delivered massive improvements over the baseline system:
- F1 Score (tool selection accuracy) improved by 39.1%.
- Tool Execution Rate (reliability in the live environment) surged by 40.7%.
- Final Answer Accuracy increased by 20.3%.
Expert reviews confirmed that 90% of the automatically augmented data generated by AugFC was directly usable without any manual corrections.
Limitations and Future Work
The current iteration of this pipeline is optimized primarily for single-hop function calling, where queries map to independent tool executions. In complex financial workflows, users often require multi-step reasoning where the output of one API serves as the input for another. The authors note that future work will focus on extending this data-driven framework to handle complex multi-tool dependencies and cross-module call scenarios.
Summary updated