RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs

Vibha Belavadi, Tushar Vatsa, Dewang Sultania, Suhas Suresha, Ishita Verma, Cheng Chen, Tracy Holloway King, Michael FriedrichNAACL 2025

TL;DR

This paper introduces RouteNator, a novel multi-modal architecture for generating synthetic training data for Large Language Models (LLMs) performing function calling. By intelligently routing requests through specialized LLM prompts and leveraging domain knowledge, RouteNator produces more diverse and realistic data, significantly improving LLM performance on function calling tasks compared to existing methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used to interpret natural language queries and translate them into executable function calls, especially in complex applications like digital content creation tools. However, training these LLMs effectively requires vast amounts of task-specific data. Real-world user interaction data is often scarce, and privacy concerns prevent its direct use. Existing synthetic data generation methods fall short, producing data that lacks diversity and complexity, failing to capture real-world distributions and leading to suboptimal LLM performance. This paper tackles the challenge of generating high-quality, diverse, and realistic synthetic training data for function calling LLMs, particularly when real-world data is unavailable.

Methods

The core of RouteNator is a router-based multi-modal architecture. This system intelligently directs user queries to different generation paths.

  • Weighted Router: A key innovation is a "weighted router" that uses population-level statistics to decide which prompt template to use. This ensures the generated data mimics real-world query distributions.
  • Specialized Prompt Templates: The architecture employs multiple specialized LLM prompt templates, each designed for specific query types (e.g., based on length, API type, content requirements).
  • Multi-Modal Integration: RouteNator integrates text-to-text LLMs with vision-to-text LLMs (specifically InternVL) to generate data that incorporates visual aspects, increasing diversity and realism.
  • Domain Knowledge Integration: The system leverages structured domain knowledge, including content metadata and knowledge graphs, to create contextually relevant queries.
  • Iterative Refinement: The authors explored several approaches, starting with template-based heuristic generation and single-prompt LLM generation, before arriving at the more sophisticated router-based multi-modal architecture.

Datasets

  • Golden Dataset: The authors created a manually curated golden dataset of 460 real-world user queries with high-quality labels for evaluation. This dataset includes 237 "Search" queries and 223 "Generate" queries, annotated with function classification, content type, and optimized sub-prompts.
  • Synthetic Datasets: The paper compares performance across datasets generated by three approaches:
  • Heuristic-based
    • Single-Prompt LLM Based
    • Router-Based (the proposed method)

Key Insights and Ideas

  • The "Router" is Crucial for Realism: Simply using LLMs to generate data isn't enough. A routing mechanism that considers real-world usage patterns (like query length and type distribution) is essential for creating effective training data.
  • Multi-Modality Boosts Diversity: Incorporating vision-to-text models allows the generation of queries that capture visual nuances, which is critical for content-related APIs and leads to more diverse training data.
  • Domain Knowledge is Key: Integrating structured knowledge (like knowledge graphs) helps generate semantically coherent and contextually relevant queries, moving beyond generic instruction tuning.
  • Balancing Diversity and Realism: The challenge lies in creating data that is both diverse enough to cover various scenarios and realistic enough to reflect how users actually interact with systems. RouteNator aims to strike this balance.

Key Results

  • Improved Function Calling Accuracy: LLMs fine-tuned with RouteNator's synthetic data consistently outperformed models trained on traditional synthetic data. For the Gorilla model, fine-tuning on the router-based synthetic dataset alone achieved an F1-Score of 0.844, compared to 0.788 for single-prompt data and 0.801 for heuristic data.
  • Enhanced Content Type Accuracy and Subprompt Similarity: The router-based approach also showed improvements in Content Type Accuracy (CTA) and Subprompt Similarity (SS).
  • Balanced Content Type Distribution: RouteNator generated synthetic data with a more even distribution across content types (e.g., Templates, Images, Videos) compared to traditional methods, which tend to overrepresent certain types like "Image."
  • Reduced Keyword Position Bias: The proposed method creates more varied keyword positions within queries, preventing models from over-fitting to fixed patterns.
  • Realistic Query Length Distributions: The system generates "Search" queries that are generally shorter and "Generate" queries that can be longer, mirroring real-world user behavior.

Limitations and Future Work

  • Limitations: While RouteNator shows significant gains, the paper notes that further improvement is possible in content-type classification. The current approach focuses on English queries.
  • Future Work:
    • Multilingual Support: Extend the system to handle multilingual query processing for broader accessibility.
  • Domain Generalization: Adapt the principles of RouteNator to other domains requiring sophisticated function-calling mechanisms.
  • Leveraging Larger Models: Explore using even more advanced LLMs for synthetic data generation to potentially yield higher-quality examples.
  • Dataset Expansion: Expand the golden dataset for more rigorous model evaluation.
  • Broader Functionality: Investigate extensibility to support additional specialized functions and API calls.

Summary updated

BibTeX
@inproceedings{belavadi2025routenator,
  author = {Vibha Belavadi and Tushar Vatsa and Dewang Sultania and Suhas Suresha and Ishita Verma and Cheng Chen and Tracy Holloway King and Michael Friedrich},
  title = {{RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure