Scalable and Cost Effective High-Cardinality Classification with LLMs via Multi-View Label Representations and Retrieval Augmentation

Anup Pattnaik, Sasanka Vutla, Hamvir Dev, Jeevesh Nandan, Cijo GeorgeEMNLP 2025

TL;DR

This paper tackles the challenge of classifying text into a very large number of categories (high cardinality), a common problem in contact centers. They propose a novel two-stage approach that first retrieves a small subset of relevant labels and then uses a Large Language Model (LLM) for final classification, significantly improving accuracy and reducing costs compared to existing methods.

What Problem Does This Paper Address?

Contact centers generate vast amounts of interaction data (calls, chats) that need to be classified into numerous categories for analysis, quality assurance, and business insights. This "high-cardinality classification" is difficult because:

  • Scalability: Traditional methods require training separate models for each domain or label set, which is expensive and time-consuming for businesses with diverse needs.
  • Dynamic Nature: Business requirements change, necessitating frequent model retraining or redevelopment.
  • Data Curation: Creating high-quality labeled datasets for specialized domains is a persistent bottleneck.
  • LLM Limitations: While LLMs offer flexibility, their performance degrades with a large number of labels, becoming sensitive to label order and prone to inconsistencies.

The authors aim to create a flexible, scalable, and cost-effective solution that leverages LLMs without the drawbacks of traditional approaches or vanilla LLM prompting.

Methods

The core of their solution is a two-stage Retrieval-Augmented Classification (RAC) framework:

  1. Stage 1: Candidate Label Retrieval:
    • Input Pre-processing: For long texts, a summarization step is applied to retain key information relevant for classification.
  2. Multi-View Label Representations: Instead of just using label names, they create three "views" for each label:
  3. Name View (NV): The label's name itself.
  4. Description View (DV): An LLM-generated description of the label.
  5. Examples View (EV): Representative examples of text that belong to that label.
  6. Vector Embeddings: Each of these views is converted into vector embeddings.
  7. Similarity Scoring: A composite similarity score is calculated for a given input text against each label, using a weighted combination of the similarity scores from the three views.
  8. Top-K Retrieval: This stage filters the entire label space down to a manageable subset of K most relevant candidate labels. This significantly reduces the search space for the LLM.
  9. Stage 2: Final Classification:
    • The LLM receives the input text and the K retrieved candidate labels (along with their descriptions).
  10. The LLM performs zero-shot classification on this reduced set, making the final prediction.

This two-stage process aims to reduce computational complexity from O(|L|) to O(K), where K << |L|, while maintaining high accuracy.

Datasets

The authors evaluated their approach on a combination of proprietary and public datasets:

  • Proprietary Contact Center Datasets: Four datasets from distinct domains: Insurance, E-Commerce, Travel, and Debt Collection. These datasets have high label cardinality, ranging from 77 to 800 labels.
  • Open Datasets:
    • Standard: CLINC150 and Banking77 (single-label, flat hierarchy).
  • Adapted: Leaf labels from hierarchical datasets: Amazon Reviews, Web of Science, and DBPedia. The parent hierarchy information was not used.

Key Insights and Ideas

  • Multi-View Label Representations are Crucial: Simply using label names is insufficient for high-cardinality tasks. Combining label names, LLM-generated descriptions, and representative examples significantly improves retrieval accuracy.
  • Representative Examples Provide Strongest Signal: While all views are beneficial, concrete examples tend to offer the most discriminative power for classification.
  • Descriptions Disambiguate: Label descriptions are particularly valuable in conversational domains where label names can be ambiguous, leading to substantial accuracy gains.
  • Retrieval Reduces LLM Cognitive Load: By pre-filtering labels, the RAC approach allows LLMs to focus on a smaller, more relevant set, leading to more consistent and accurate predictions, especially in high-cardinality scenarios.
  • Cost Savings are Significant: The retrieval step dramatically reduces the number of tokens processed by the LLM, leading to substantial cost reductions (up to 91%) without sacrificing accuracy.

Key Results

  • Accuracy Gains: The proposed RAC method achieved accuracy improvements of up to 14.6% compared to baseline LLM approaches across various models (Nova-Lite, GPT-40-mini, Llama 4 Maverick 17B).
  • Consistency Improvement: The method significantly reduces prediction variance, with improvements of over 9.4% compared to full-label prompting.
  • Cost Reduction: Inference costs were reduced by 60-91%, with an average of 72.36%, due to the reduced number of tokens processed by the LLM.
  • Robustness: The approach demonstrated consistent performance gains across diverse proprietary and open-source datasets, highlighting its generalizability.
  • Optimal Top-K: The study found that optimal Top-K values vary by dataset, and a tuned approach generally outperforms fixed K values, balancing retrieval coverage with classification complexity.

Limitations and Future Work

  • Retrieval Dependence: The pipeline's effectiveness hinges on accurate label retrieval. Retrieval failures (up to 10.66% in some settings) are unrecoverable by the classification stage.
  • View Weight Sensitivity: Optimal weights for label views vary by dataset, requiring tuning for new domains.
  • Prompt Sensitivity: While improved, LLM outputs can still retain some sensitivity to prompt structure, especially with larger K.
  • Monolingual Focus: The current work is limited to English. Extending to multilingual scenarios is an open area.
  • Separate Optimization: Retrieval and classification are optimized independently. Joint end-to-end optimization could potentially yield better performance but might reduce flexibility.
  • Off-the-shelf Embeddings: Reliance on general-purpose embedding models might not capture domain-specific nuances optimally. Future work could explore learnable re-rankers or cross-encoders for better label-text relationship capture.

Summary updated

BibTeX
@inproceedings{pattnaik2025scalable,
  author = {Anup Pattnaik and Sasanka Vutla and Hamvir Dev and Jeevesh Nandan and Cijo George},
  title = {{Scalable and Cost Effective High-Cardinality Classification with LLMs via Multi-View Label Representations and Retrieval Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure