DispatchQA: A Benchmark for Small Function Calling Language Models in E-Commerce Applications

Joachim Daiber, Victor Maricato, Ayan Sinha, Andrew RabinovichEMNLP 2025

TL;DR

This paper introduces DispatchQA, a new benchmark designed to evaluate how well small language models (SLMs) can translate natural language e-commerce search queries into structured API calls, focusing on both search relevance and inference latency. They demonstrate that fine-tuned SLMs can match or even surpass larger models in search quality while offering significantly lower latency, making them a practical choice for real-world e-commerce applications.

What Problem Does This Paper Address?

Modern e-commerce relies heavily on natural language search, but understanding the nuances of user queries remains a significant challenge for traditional systems. While large language models (LLMs) have shown promise in interpreting user intent and translating it into executable actions via function calling, deploying these large models in production e-commerce environments is hindered by practical concerns like latency and cost. E-commerce search systems require responses under 500ms to maintain user engagement. Small language models (SLMs) offer a potential solution due to their lower latency and cost, but the trade-offs between their function-calling capabilities and latency in this specific domain were not well understood. This paper aims to fill that gap by providing a benchmark and evaluation framework tailored to this problem.

Methods

The authors introduce DispatchQA, a benchmark that simulates e-commerce search and evaluates SLMs' ability to perform function calling. The core of their methodology involves:

  • Function Calling Paradigm: Natural language queries are translated into structured API calls (specifically, a search API with parameters like query, color, price, etc.) that a downstream search system can execute.
  • LLM-as-a-Judge: To automate relevance evaluation, they employ a large language model (GPT-4o) to act as a judge, assessing the relevance of search results to user queries. They validate this approach by showing high agreement (κ = 0.71 on WANDS, κ = 0.65 on WebShop) with human judgments.
  • Realistic Simulation: The benchmark uses a simulation based on the WebShop dataset, incorporating a traditional BM25 search system.
  • Holistic Evaluation: The evaluation methodology measures both search result quality (using standard IR metrics like NDCG, MAP, MRR) and end-to-end latency.
  • Baseline Models and Training Recipes: They provide strong, replicable baselines, including fine-tuning recipes for Llama 3.1 8B Instruct, to facilitate standardized comparisons.

Datasets

  • WebShop Dataset: The benchmark is based on the WebShop dataset, which contains product information from Amazon.com. They selected a subset of attributes (price, rating, category, country of origin, brand, manufacturer, color, material, style) for their search schema.
  • WANDS Dataset: Used for validating the LLM-as-a-judge system's reliability.
  • Synthetic Training Data: They generated a synthetic dataset of 1000 (query, function-call) pairs for fine-tuning SLMs. This data was created using a "neighborhood-based sampling" strategy where a teacher model (GPT-4o) generates queries and parameters based on a group of similar products.

Key Insights and Ideas

  • SLMs are Viable for E-commerce Search: Fine-tuned SLMs can achieve search quality comparable to or better than much larger commercial LLM APIs, making them a practical and efficient choice for production.
  • Latency is Crucial: The benchmark emphasizes end-to-end latency as a critical metric, highlighting that even small differences in inference speed can significantly impact user experience.
  • Structured Decoding Matters: The choice of structured decoding strategy (e.g., using frameworks like SGLang) has a substantial impact on both latency and quality. Some methods, like JSONformer, were found to be slow and less effective.
  • Prompting Strategies Influence Performance: Including the full JSON schema in the prompt can increase latency for closed models with minimal quality gains, while for local SLMs, it can be detrimental. Task-specific fine-tuning can internalize this knowledge, reducing the need for explicit schema repetition.

Key Results

  • Quality vs. Latency Trade-off: Fine-tuned Llama 3.1 8B Instruct models achieved search relevance on par with GPT-40, but with up to 3x faster inference.
  • LLM-as-a-Judge Reliability: The LLM-as-a-judge system demonstrated substantial agreement (κ = 0.71) with human judgments, validating its use for large-scale evaluations.
  • Structured Decoding Performance: SGLang implementations of structured decoding provided high-quality, low-latency results, outperforming JSONformer significantly in both speed and relevance.
  • Prompting Impact: For local SLMs, omitting the JSON schema from the prompt led to better latency and quality compared to including it.

Limitations and Future Work

  • Limited Model Scope: The study primarily focused on fine-tuning one base SLM (Llama 3.1 8B Instruct) and evaluating a limited set of closed LLM APIs. Future work could expand this to include more systems.
  • E-commerce Domain Specificity: The benchmark and dataset are specific to e-commerce and based on Amazon.com data. Generalizing findings to other domains requires further evaluation.
  • Language Specificity: While models are not inherently language-specific, the queries used are in English, limiting direct applicability to other languages without further testing.
  • Data Noise: The WebShop dataset, derived from Amazon.com, may contain noise and variability, which could affect benchmark reliability. Future iterations aim to incorporate additional datasets.
  • Reproducibility of Latency: Achieving perfectly consistent latency estimates remains challenging due to hardware, network, and caching fluctuations. The authors took steps to mitigate this but acknowledge residual variance.

Summary updated

BibTeX
@inproceedings{daiber2025dispatchqa,
  author = {Joachim Daiber and Victor Maricato and Ayan Sinha and Andrew Rabinovich},
  title = {{DispatchQA: A Benchmark for Small Function Calling Language Models in E-Commerce Applications}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure