Data-Driven Function Calling Improvements in Large Language Model for Online Financial QA

Xing Tang, Hao Chen, Shiwei Li, Fuyuan Lyu, Weijie Shi, Lingjie Li, Dugang Liu, Weihong Luo, Xiku Du, Xiuqiang HearXiv 2026 · arXiv

TL;DR

This paper introduces a data-driven pipeline designed to enhance the function-calling capabilities of Large Language Models (LLMs) within live financial question-answering (QA) systems. By automatically detecting dataset "blind spots" using information entropy and employing a flexible two-step training process (SFT + RL), the proposed framework significantly boosts tool-use accuracy and reliability. The pipeline has been successfully deployed in production on Tencent's YuanBao platform, one of China's largest chat applications.

What Problem Does This Paper Address?

Deploying LLMs for real-world financial QA is notoriously difficult. Financial data changes rapidly, requiring models to interface with highly customized, private APIs to fetch real-time information. Generic LLMs struggle with these specialized tools because they lack domain-specific training data.

Furthermore, production environments introduce massive query diversity. Users rarely format queries perfectly; they use company nicknames (e.g., calling Tencent "Goose Factory" or using its stock ticker "700") and input highly varied parameters. Because real-world user queries typically follow a power-law distribution, standard training datasets suffer from parameter distribution collapse—meaning a few common parameter values dominate the data, leaving "blind spots" where the model fails to generalize to rarer, long-tail inputs.

Figure 1: The overview of the key points and corresponding challenges in the online financial QA system powered by LLM.

Methods

To overcome these challenges, the authors developed an automated, data-driven pipeline consisting of three core phases:

Figure 2: The data-driven pipeline consisting of data collection, data construction, data augmentation, and model training.
  1. Data Collection & Construction: The system continuously captures live user queries. To ensure data quality without overwhelming human annotators, queries are validated via embedding models and passed to a powerful teacher LLM for few-shot tool-call generation. Human experts are called in only to resolve inconsistencies between the production model and the teacher model.
  2. Automated Data Augmentation (AugFC): To eliminate parameter blind spots, the authors introduce AugFC. This method uses semantic clustering (K-Means) to group queries and calculates both global and local information entropy for parameter values. If a parameter's local entropy within a cluster is significantly lower than its global entropy, it is flagged as a "blind spot" (indicating value collapse). The system then prompts an LLM to generate distribution-aware counterfactual queries to fill these specific gaps.
  3. Two-Step Model Training:
  4. Supervised Fine-Tuning (SFT) with Prompt Isolation: The model is trained on a mix of reasoning-heavy samples (with Chain-of-Thought planning) and direct-calling samples. By isolating these styles into two distinct system prompts, developers can toggle off the reasoning tokens during low-latency inference modes to save computational overhead.
  5. Reinforcement Learning (RL): The model is optimized using Group Relative Policy Optimization (GRPO). It relies on a rule-based reward function that scores the output based on format compliance, correct tool selection, parameter key retrieval, and exact parameter value matching.

Datasets

The pipeline was evaluated across both offline benchmarks and live production data:

  • Offline Benchmarks: API-Bank (Levels 1 and 2), Tool-Alpaca, Seal-Tools, Nexus Raven Evaluation, and a sampled subset of xLAM-small.
  • Online Production Dataset: A proprietary dataset derived from live interactions on Tencent's YuanBao platform, evaluated using 350 end-to-end query-answer pairs comprising nearly 2,000 tool-call instances.

Key Insights and Ideas

  • The Power-Law Trap: Real user interactions naturally cluster around a tiny fraction of possible parameter values. Relying solely on raw user logs for fine-tuning inadvertently trains models that fail on rare but critical API arguments.
  • Entropy as a Blind-Spot Radar: Instead of guessing where a dataset lacks diversity, calculating the ratio of local-to-global information entropy mathematically pinpoints exactly which parameters have collapsed within specific semantic contexts.
  • Prompt Isolation for Latency Control: Chain-of-Thought (CoT) reasoning improves tool-calling accuracy but introduces severe token latency. Training the model to handle dual system prompts (one with a <plan> tag and one without) allows production systems to dynamically balance accuracy and speed depending on the query type.

Key Results

Offline Performance

When applied to base models like Qwen2.5 (ranging from 1.5B to 32B parameters), the combined dataset (src+aug) consistently outperformed models trained purely on seed data. The two-step training method also beat out established function-calling models like xLAM and Hammer across the six offline benchmarks.

Online Production Gains

In live deployment on the YuanBao platform, the data-driven pipeline delivered massive improvements over the baseline system:

  • F1 Score (tool selection accuracy) improved by 39.1%.
  • Tool Execution Rate (reliability in the live environment) surged by 40.7%.
  • Final Answer Accuracy increased by 20.3%.

Expert reviews confirmed that 90% of the automatically augmented data generated by AugFC was directly usable without any manual corrections.

Figure 4: The number of blind spots to repair with different LLMs.

Limitations and Future Work

The current iteration of this pipeline is optimized primarily for single-hop function calling, where queries map to independent tool executions. In complex financial workflows, users often require multi-step reasoning where the output of one API serves as the input for another. The authors note that future work will focus on extending this data-driven framework to handle complex multi-tool dependencies and cross-module call scenarios.

Summary updated

Abstract

Large language models (LLMs) have been incorporated into numerous industrial applications. Meanwhile, a vast array of API assets is scattered across various functions in the financial domain. An online financial question-answering system can leverage both LLMs and private APIs to provide timely financial analysis and information. The key is equipping the LLM model with function calling capability tailored to a financial scenario. However, a generic LLM requires customized financial APIs to call and struggles to adapt to the financial domain. Additionally, online user queries are diverse and contain out-of-distribution parameters compared with the required function input parameters, which makes it more difficult for a generic LLM to serve online users. In this paper, we propose a data-driven pipeline to enhance function calling in LLM for our online, deployed financial QA, comprising dataset construction, data augmentation, and model training. Specifically, we construct a dataset based on a previous study and update it periodically, incorporating queries and an augmentation method named AugFC. The addition of user query-related samples will \textit{exploit} our financial toolset in a data-driven manner, and AugFC explores the possible parameter values to enhance the diversity of our updated dataset. Then, we train an LLM with a two-step method, which enables the use of our financial functions. Extensive experiments on existing offline datasets, as well as the deployment of an online scenario, illustrate the superiority of our pipeline. The related pipeline has been adopted in the financial QA of YuanBao\footnote{https://yuanbao.tencent.com/chat/}, one of the largest chat platforms in China.
BibTeX
@misc{tang2026data,
  author = {Xing Tang and Hao Chen and Shiwei Li and Fuyuan Lyu and Weijie Shi and Lingjie Li and Dugang Liu and Weihong Luo and Xiku Du and Xiuqiang He},
  title = {{Data-Driven Function Calling Improvements in Large Language Model for Online Financial QA}},
  year = {2026},
  eprint = {2604.05387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.05387},
}
Figure