Can Decision Models Replace LLMs in Real-World Contact Center Classification and Judging Tasks?

Md Tahmid Rahman Laskar, Xue-Yong Fu, Shashi Bhushan TnarXiv 2026 · arXiv

In the world of AI-powered contact centers, generative Large Language Models (LLMs) are the current heavyweights, handling everything from summarization to customer intent classification. But using a massive generative model to answer a multiple-choice question is like using a sledgehammer to crack a nut—it’s expensive and computationally wasteful.

A recent paper from researchers at Dialpad, titled "Can Decision Models Replace LLMs in Real-World Contact Center Classification and Judging Tasks?", explores a leaner alternative: Decision Models. These models are designed to return a specific typed answer and a probability score directly, bypassing the need for free-form text generation.

TL;DR

Decision models (like Gemini-Decision-1 and JEV) offer a high-performance, cost-effective alternative to generative LLMs for bounded tasks like intent classification. While they match frontier LLMs in accuracy and provide reliable confidence scores for automated routing, they struggle with "no-answer" scenarios and exhibit significant position bias when acting as judges.

What Problem Does This Paper Address?

Contact centers rely on "bounded decisions"—tasks where the answer must come from a pre-defined set of options. Examples include identifying a caller's intent, checking if an agent followed a specific script, or selecting the right knowledge-base article.

Generative LLMs are increasingly used for these tasks, but they introduce substantial inference costs and latency. "Decision models" provide a specialized interface: you give them a context, a question, and a list of allowed options, and they return the most likely option along with a probability distribution. The authors set out to determine if these specialized models can truly replace general-purpose LLMs in real-world, zero-shot production environments.

Methods

The researchers conducted a massive head-to-head comparison:

  • The Contenders: 9 decision models (including API-based models like JEV and Gemini-Decision-1, and open-weight models like Clef and OpenJev) vs. 13 generative LLMs (Qwen, Gemma, GPT, Gemini, and Claude series).
  • The Setup: All models were evaluated in a zero-shot setting—meaning no task-specific fine-tuning or few-shot examples were provided.
  • The Evaluation: The study covered eight real-world contact center tasks, split into six classification tasks and two "LLM-as-a-judge" tasks.
Figure 1: Overview of the evaluation. Decision models and generative LLMs answer the same instances zero-shot.

Datasets

The study utilized proprietary, PII-redacted data from Dialpad, representing actual business conversations:

  • Classification Tasks: Call Outcome (30 labels), Purpose of Call (30 labels), Intent (4–21 options), Quality Assurance (Yes/No), Procedural Step (Yes/No), and RAG Document Selection.
  • Judging Tasks: Agent Workflow Judging (evaluating if an AI agent followed a complex skill workflow) and Generated Response Judging (pointwise and pairwise evaluation of model summaries).

Key Insights and Ideas

  • The Abstention Gap: The biggest weakness of decision models is their reluctance to say "I don't know." In tasks like Intent classification, most decision models rarely chose the "no-answer" option, even when none of the provided intents fit. Gemini-Decision-1 was the notable exception, performing significantly better at abstaining than its peers.
  • Confidence as a Routing Signal: Unlike generative LLMs, decision models provide native probability scores. The researchers found that when a decision model is highly confident ($p \ge 0.9$), it is remarkably accurate (often >95%). This allows engineers to "selectively route" confident predictions for automation while sending low-confidence cases to a human or a stronger model.
Figure 2: Accuracy of the five strongest classification decision models on all predictions (empty markers) and on confident predictions with selected-option probability p ≥0.9 (filled markers), for each classification task. The number above each pair is the percentage of predictions that are confident.
  • Judging is Harder than Classifying: While decision models held their own in classification, their performance plummeted in judging tasks. Smaller models, in particular, failed to align with human/frontier-LLM judgments and showed a strong position bias (a tendency to favor the first or second option regardless of content).

Key Results

  • Classification Performance: The generative Qwen-3.8-27B was the overall leader with 78.6% average accuracy, but Gemini-Decision-1 followed closely at 76.2%, outperforming many larger generative models.
  • Cost Efficiency: API-based decision models are significantly cheaper. For example, JEV costs roughly $0.09 per 1,000 requests**, compared to **$1.73 for a generative model like Gemini-3.8-Flash.
  • Speed: Open-weight 9B decision models were the fastest, with a median latency of 0.4 seconds on a single A100 GPU.
Main accuracy results across six classification tasks.

Limitations and Future Work

The authors acknowledge several limitations:

  1. Offline Nature: The study was conducted on stored data and does not measure live customer outcomes.
  2. Synthetic Data: The "Intent" utterances were synthetic, which may not perfectly capture the messiness of real human speech.
  3. Language & Scope: The evaluation was limited to English and specific contact center use cases.

Future work will focus on developing better "calibrated abstention" thresholds—essentially teaching these models when to stay silent—and expanding the evaluation to more diverse, multi-lingual tasks. For now, the takeaway is clear: if you have a bounded classification task, a decision model might just be the most efficient tool for the job.

Summary updated

Abstract

Contact center systems often need to make bounded decisions, such as identifying a caller's intent, retrieving the knowledge-base article that answers a query, assessing agent compliance, or judging the output of AI models used in production. Decision models return a typed answer and its probability directly, making them a promising cost-effective alternative to generative Large Language Models (LLMs) in such bounded conversational tasks. However, their effectiveness in real-world contact center applications remains unclear. In this paper, we compare nine decision models against thirteen generative LLMs in zero-shot settings across eight contact center use cases: six classification tasks and two judging tasks. Our experiments demonstrate that many decision models can match the performance of frontier LLMs on contact center classification, and API-based closed decision models achieve this at the lowest per-request cost among the API models we evaluate. We further observe that their confident predictions are more accurate, enabling selective routing in real-world production scenarios. Across classification tasks, the main weakness of most decision models is abstention: they rarely choose the no-answer option, which accounts for most of their gap to the strongest LLMs (although Gemini-Decision-1 is the notable exception in this regard). We also find that the judging quality of decision models drops sharply for smaller models, and they are also more prone to position bias. In short, our findings provide practical guidance for selecting decision models based on abstention behavior, confidence, and deployment cost, rather than overall accuracy alone.
BibTeX
@misc{laskar2026can,
  author = {Md Tahmid Rahman Laskar and Xue-Yong Fu and Shashi Bhushan TN},
  title = {{Can Decision Models Replace LLMs in Real-World Contact Center Classification and Judging Tasks?}},
  year = {2026},
  eprint = {2611.22949},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22949},
}
Figure