STREAQ: Selective Tiered Routing for Effective and Affordable Contact Center Quality Assurance

Prajwal Sood, Rajdeep Agrawal, Mayank Sati, Digvijay Anil Ingle, Cijo GeorgeEMNLP 2025

TL;DR

This paper introduces STREAQ, a novel two-tier routing system that intelligently directs contact center conversations to either a cost-effective or a high-capability AI model. This approach significantly reduces operational costs while maintaining high accuracy in quality assurance tasks, making enterprise-scale automated QA more practical and affordable.

What Problem Does This Paper Address?

Contact centers handle millions of customer interactions daily, and ensuring quality through agent evaluation is crucial. Traditional manual Quality Assurance (QA) struggles to scale with this volume. While advanced AI models, particularly Large Language Models (LLMs), can automate QA, deploying them at scale is prohibitively expensive. Smaller, cheaper LLMs are more affordable but lack the precision needed for critical "Answers of Interest" (AoIs), leading to high false positive rates and eroding agent trust. This creates a dilemma: high cost for high accuracy, or low cost with low accuracy. The paper tackles this by finding a middle ground: using AI to decide which AI to use for each evaluation.

Methods

The core innovation is STREAQ, a two-tier selective routing framework.

  1. Tier 1 (Small LM): A cost-efficient, smaller LLM (like the in-house model) processes the conversation and generates an initial evaluation.
  2. Tier 2 (Large LM): A more capable, expensive LLM is only invoked if the first tier's assessment is deemed uncertain or potentially incorrect.

The "intelligence" lies in the routing function, which decides when to escalate. The paper explores a learned routing strategy using a fine-tuned RoBERTa-based binary classifier. This classifier learns to identify low-confidence predictions from the smaller model and routes them to the larger model. Crucially, the authors found that passing the reasoning trace from the smaller model to the larger one can actually degrade performance, suggesting that the learned router should focus on the prediction rather than the reasoning process of the cheaper model.

Datasets

The paper uses a proprietary contact center dataset called DQA. This dataset consists of 1,879 question-conversation pairs, covering various evaluation categories like customer experience, compliance, and professionalism. The data was curated and annotated by five domain experts, with ground truth labels established by majority agreement.

Key Insights and Ideas

  • The "Cost-Scale Dilemma" is Real: Deploying top-tier LLMs for every QA task is economically infeasible for large contact centers.
  • Reasoning Trace Can Be Harmful: Surprisingly, feeding the detailed reasoning of a weaker LLM to a stronger one can lead to worse overall performance. The stronger model can become biased by the weaker model's flawed logic.
  • Selective Routing is Key: STREAQ offers a practical solution by intelligently routing conversations, balancing cost and accuracy.
  • Focus on the Prediction, Not the Process: The most effective routing strategy learned to make decisions based on the output of the smaller model, not its internal reasoning steps.

Key Results

  • Significant Cost Reduction: STREAQ, using Nova-Pro as the cheaper model and Claude-4-Sonnet as the expensive one, achieved up to a 48% cost reduction while preserving 88.9% of the full model's AoI precision. This translates to a daily cost saving of approximately $17,842.
  • High AoI Precision Maintained: The system managed to retain high precision on "Answers of Interest" (AoI), which are critical but rare events, a key challenge in skewed datasets. For example, using Nova-Pro in a two-tier setup with Claude-4-Sonnet retained 76% AoI Precision while significantly cutting costs.
  • Ablation Studies Validate Design: Experiments showed that omitting the smaller model's reasoning trace (Setup P4) led to substantial improvements in AoI metrics compared to including it.

Limitations and Future Work

  • Dataset Specificity: The evaluation was conducted on a proprietary, English-language dataset, limiting generalizability to other languages or domains.
  • Model Family Focus: The study evaluated a specific set of LLMs; performance might vary with newer models or architectures.
  • Binary QA Tasks: The evaluation focused on binary (yes/no) QA questions, which may not capture the full complexity of real-world assessments.
  • API Pricing Fluctuations: Cost analysis is based on current API pricing, which is subject to change.

The authors suggest that future work could explore more sophisticated routing strategies, investigate the impact of different LLM families, and extend the framework to more complex, multi-faceted QA tasks. They also highlight the need for continuous ethical consideration and adaptation as AI in performance evaluation evolves.

Summary updated

BibTeX
@inproceedings{sood2025streaq,
  author = {Prajwal Sood and Rajdeep Agrawal and Mayank Sati and Digvijay Anil Ingle and Cijo George},
  title = {{STREAQ: Selective Tiered Routing for Effective and Affordable Contact Center Quality Assurance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure