Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline

Guancheng Zeng, Wentao Ding, Beining Xu, Chi Zhang, Wenqiang Han, Gang Li, Jingjing Mo, Pengxu Qiu, Xinran Tao, Wang Tao, Haowen HuarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces a specialized pipeline for training Large Language Models (LLMs) to accurately call enterprise-specific APIs. By focusing on data synthesis, targeted fine-tuning with LoRA, and robust evaluation, they demonstrate significant improvements in function-calling capabilities, even outperforming state-of-the-art models like GPT-4 on specific enterprise tasks. This work is crucial for enabling AI agents to effectively leverage existing enterprise API infrastructure.

What Problem Does This Paper Address?

Enterprises have vast collections of APIs that form the backbone of their business processes. AI agents, powered by LLMs, can leverage these APIs as functional tools to automate workflows and enhance efficiency. However, general-purpose LLMs often struggle to meet enterprise requirements for accuracy, efficiency, and stability when it comes to function-calling. This is largely due to a lack of domain-specific training data and the unique complexities of enterprise scenarios. Generic models may fail to correctly identify the right tool, extract parameters, or format the function call, leading to errors and unreliable agent behavior. This paper addresses the need for a tailored approach to train LLMs for precise function-calling within specific enterprise contexts, especially for Small and Medium Enterprises (SMEs) with limited computational resources.

Methods

The core of this work is a specialized training pipeline designed for enterprise-scenario function-calling capabilities. It comprises three main modules:

  1. Data Synthesis Module: This module focuses on generating high-quality, scenario-specific data. It starts with "seed questions" (either AI-generated or human-annotated) and then employs data augmentation strategies (replacement, rewriting, simplification, error introduction) to create a diverse dataset. This process aims to maximize data quality and quantity, ensuring it aligns with real-world business scenarios and tool functionalities.
  2. Model Fine-Tuning Module: This module utilizes the synthesized data to fine-tune a generic function-calling LLM. The authors employ the LoRA (Low-Rank Adaptation) method, which is parameter-efficient and suitable for resource-constrained environments. This allows for adapting pre-trained models without the need for extensive computational power.
  3. Model Evaluation Module: This module assesses the fine-tuned model's performance using AST (Abstract Syntax Tree) evaluation for structural correctness and accuracy of function selection and parameter filling. It also uses confusion matrices to analyze tool selection errors, providing insights into model behavior and areas for improvement.

Datasets

The authors created a specialized dataset for a digital HR intelligent agent scenario. This dataset includes:

  • AI-generated samples: 1,260 fully AI-synthesized function-calling training samples.
  • Manually augmented samples: 1,035 augmented samples derived from 207 manually annotated seed instances.
  • Combined dataset (DHR_train_3): A total of 2,295 instances, combining AI-generated and human-augmented data.

For evaluation, they used two test sets:

  • DHR_test_A: 207 manually annotated seed questions.
  • DHR_test_B: 135 distinct questions to prevent overfitting bias.

The paper does not introduce a new benchmark but leverages existing evaluation frameworks like BFCL.

Key Insights and Ideas

  • The Power of Scenario-Specific Data: Generic LLMs, while capable, fall short in specialized enterprise contexts. Tailoring training data to specific API functionalities and user query patterns is paramount for achieving high accuracy and reliability.
  • LoRA for Efficient Adaptation: LoRA proves to be an effective and resource-efficient method for fine-tuning LLMs for function-calling tasks, making advanced AI agent development more accessible to organizations with limited computational budgets.
  • Data Augmentation is Key: A combination of AI-generated and human-annotated data, coupled with diverse augmentation strategies, significantly boosts model performance. This approach balances the scale of AI generation with the quality and relevance of human expertise.
  • AST Evaluation for Structured Tasks: For function-calling, AST-based evaluation is superior to execution-based methods when dealing with potentially fictitious parameters or when rapid, offline verification is needed.
  • Tool Description Matters: The length and clarity of tool descriptions influence model performance. Shorter, well-defined descriptions can lead to better generalization and accuracy, striking a balance between specificity and broad applicability.

Key Results

The fine-tuned 7B-parameter model (Qwen2.5-Coder-7B-Instruct) using the proposed pipeline demonstrated impressive results:

  • Superior Performance: The DHR_train_3_ft model achieved 97.6% structural completeness, 97.6% tool selection accuracy, and 100% parameter filling accuracy on the DHR_test_A test set.
  • Outperforming SOTA: The fine-tuned model surpassed GPT-4 and GPT-40 in accuracy on the test sets, particularly in structural completeness and parameter filling.
  • Error Reduction: Significant improvements were observed across all metrics (structural completeness, tool selection, parameter filling) after fine-tuning. The model achieved complete accuracy in output structure and parameter filling.
  • Enhanced Tool Selection: The model showed improved differentiation between similar tools and enhanced F1 scores for error-prone tools, indicating a comprehensive improvement in function selection capabilities.

Limitations and Future Work

  • Single-Turn Focus: The current pipeline primarily focuses on single-turn data synthesis and does not yet account for more complex multi-step or multi-turn interactions, which are common in real-world enterprise scenarios.
  • Simplistic Filtering: The data filtering process for augmented data is described as relatively simplistic, suggesting room for improvement in ensuring data quality.

Future work will aim to:

  • Enhance the filtering of augmented data to further improve data quality.
  • Extend the pipeline to handle multi-step and multi-turn interactions.
  • Integrate a data feedback module to continuously update models with user interaction data.
  • Apply the pipeline to other enterprise domains beyond digital HR.

Summary updated

BibTeX
@misc{zeng2024adaptable,
  author = {Guancheng Zeng and Wentao Ding and Beining Xu and Chi Zhang and Wenqiang Han and Gang Li and Jingjing Mo and Pengxu Qiu and Xinran Tao and Wang Tao and Haowen Hu},
  title = {{Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline}},
  year = {2024},
  eprint = {2412.15660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15660},
}
Figure