In the world of AI-powered contact centers, generative Large Language Models (LLMs) are the current heavyweights, handling everything from summarization to customer intent classification. But using a massive generative model to answer a multiple-choice question is like using a sledgehammer to crack a nut—it’s expensive and computationally wasteful.
A recent paper from researchers at Dialpad, titled "Can Decision Models Replace LLMs in Real-World Contact Center Classification and Judging Tasks?", explores a leaner alternative: Decision Models. These models are designed to return a specific typed answer and a probability score directly, bypassing the need for free-form text generation.
TL;DR
Decision models (like Gemini-Decision-1 and JEV) offer a high-performance, cost-effective alternative to generative LLMs for bounded tasks like intent classification. While they match frontier LLMs in accuracy and provide reliable confidence scores for automated routing, they struggle with "no-answer" scenarios and exhibit significant position bias when acting as judges.
What Problem Does This Paper Address?
Contact centers rely on "bounded decisions"—tasks where the answer must come from a pre-defined set of options. Examples include identifying a caller's intent, checking if an agent followed a specific script, or selecting the right knowledge-base article.
Generative LLMs are increasingly used for these tasks, but they introduce substantial inference costs and latency. "Decision models" provide a specialized interface: you give them a context, a question, and a list of allowed options, and they return the most likely option along with a probability distribution. The authors set out to determine if these specialized models can truly replace general-purpose LLMs in real-world, zero-shot production environments.
Methods
The researchers conducted a massive head-to-head comparison:
- The Contenders: 9 decision models (including API-based models like JEV and Gemini-Decision-1, and open-weight models like Clef and OpenJev) vs. 13 generative LLMs (Qwen, Gemma, GPT, Gemini, and Claude series).
- The Setup: All models were evaluated in a zero-shot setting—meaning no task-specific fine-tuning or few-shot examples were provided.
- The Evaluation: The study covered eight real-world contact center tasks, split into six classification tasks and two "LLM-as-a-judge" tasks.
Datasets
The study utilized proprietary, PII-redacted data from Dialpad, representing actual business conversations:
- Classification Tasks: Call Outcome (30 labels), Purpose of Call (30 labels), Intent (4–21 options), Quality Assurance (Yes/No), Procedural Step (Yes/No), and RAG Document Selection.
- Judging Tasks: Agent Workflow Judging (evaluating if an AI agent followed a complex skill workflow) and Generated Response Judging (pointwise and pairwise evaluation of model summaries).
Key Insights and Ideas
- The Abstention Gap: The biggest weakness of decision models is their reluctance to say "I don't know." In tasks like Intent classification, most decision models rarely chose the "no-answer" option, even when none of the provided intents fit. Gemini-Decision-1 was the notable exception, performing significantly better at abstaining than its peers.
- Confidence as a Routing Signal: Unlike generative LLMs, decision models provide native probability scores. The researchers found that when a decision model is highly confident ($p \ge 0.9$), it is remarkably accurate (often >95%). This allows engineers to "selectively route" confident predictions for automation while sending low-confidence cases to a human or a stronger model.
- Judging is Harder than Classifying: While decision models held their own in classification, their performance plummeted in judging tasks. Smaller models, in particular, failed to align with human/frontier-LLM judgments and showed a strong position bias (a tendency to favor the first or second option regardless of content).
Key Results
- Classification Performance: The generative Qwen-3.8-27B was the overall leader with 78.6% average accuracy, but Gemini-Decision-1 followed closely at 76.2%, outperforming many larger generative models.
- Cost Efficiency: API-based decision models are significantly cheaper. For example, JEV costs roughly $0.09 per 1,000 requests**, compared to **$1.73 for a generative model like Gemini-3.8-Flash.
- Speed: Open-weight 9B decision models were the fastest, with a median latency of 0.4 seconds on a single A100 GPU.
Limitations and Future Work
The authors acknowledge several limitations:
- Offline Nature: The study was conducted on stored data and does not measure live customer outcomes.
- Synthetic Data: The "Intent" utterances were synthetic, which may not perfectly capture the messiness of real human speech.
- Language & Scope: The evaluation was limited to English and specific contact center use cases.
Future work will focus on developing better "calibrated abstention" thresholds—essentially teaching these models when to stay silent—and expanding the evaluation to more diverse, multi-lingual tasks. For now, the takeaway is clear: if you have a bounded classification task, a decision model might just be the most efficient tool for the job.
Summary updated