On Assigning Product and Software Codes to Customer Service Requests with Large Language Models

Sujatha Das Gollapalli, Mouad Hakam, Mingzhe Du, See-Kiong Ng, Mohammed Hamzeh2025 · DOI

Abstract

In a technology company, quality of customer service that involves providing troubleshooting assistance and advice to customers is a crucial asset. Often, insights from historical customer service data are used to make decisions related to future product offerings. In this paper, we address the challenging problem of automatic assignment of product names and software version labels to customer Service Requests (SRs) related to Cisco Systems, Inc. We study the effectiveness of state-of-the-art Large Language Models (LLMs) in assigning the correct product name codes and software versions from several possible label options and their "noncanonical" mentions in the associated SR data. We frame this assignment as a multiple-choice question (MCQ) answering task instead of conventional prompts and devise a novel pipeline of employing a classifier for filtering inputs to the LLM for saving usage costs. On our experimental dataset based on real SRs, we are able to correctly identify product name and software version labels when they are mentioned with over 90% accuracy while cutting LLM costs by about 40-60% on average, thus providing a viable solution for practical deployment.

Introduction

Large technology enterprises like Cisco Systems manage a massive volume of unstructured data through Service Requests (SRs), which include emails, chat logs, and system outputs. A critical but tedious task for support engineers is tagging these SRs with precise product names and software version codes at the end of a case. Accurate tagging is essential for downstream analytics and product decision-making.

However, this task is fraught with challenges. Alphanumeric product codes (e.g., AIR-CT8540-1K-K9) are often referred to by non-canonical nicknames or partial identifiers in the text. Furthermore, the sheer number of possible codes in a company's catalog makes manual selection difficult and error-prone. Standard Named Entity Recognition (NER) techniques often struggle because these proprietary codes are rarely present in the general-purpose pre-training data of most LLMs.

Methodology

To address these challenges, the researchers developed a two-stage pipeline designed for both high accuracy and operational cost-efficiency.

1. The Filter Classifier

Customer service cases often contain a high volume of "noise"—notes that are administrative or conversational but contain no technical identifiers (e.g., "I will check this and get back to you"). Sending all this text to an LLM is both expensive and potentially confusing for the model. The authors fine-tuned a Flan-T5 Large model as a binary classifier to identify only the "informative" notes likely to contain product or version mentions.

2. MCQ Prompting

A key innovation of this work is framing the extraction task as a Multiple-Choice Question (MCQ). Instead of asking an LLM to "extract the product name" from scratch—which can lead to hallucinations—the system uses high-level SR metadata to look up a candidate list of valid codes from a Tech/Subtech taxonomy. The LLM is then prompted to select the most appropriate code from this restricted list.

The proposed pipeline for assigning product and software codes using a filter classifier and MCQ prompting.

3. Normalization and Matching

To handle the variations in how versions are written (e.g., 15.5(1)SY vs. 15.5.1), the pipeline incorporates custom matching logic based on Longest Common Sequence (LCS) and edit distance. This ensures that minor formatting differences do not result in false negatives.

Experimental Results

The framework was evaluated on an internal Cisco dataset comprising 1,287 SRs and over 27,000 case notes. To ensure rigorous evaluation, a "gold" subset of verified labels was used.

Prompting Strategy Impact

The results highlighted a significant performance gap between open-ended prompts (QPrompt) and MCQ-style prompts. The MCQ approach effectively bridges the gap between the LLM's general reasoning capabilities and the specific, proprietary knowledge required for Cisco products.

Performance comparison of different prompt types for Product Name (PName) and Software Version (SWV) prediction.

Key Performance Metrics

  • Accuracy: Using GPT-4o with the MCQ+NONE prompt, the system achieved 96.55% accuracy for software versions and 78.18% for product names.
  • Cost Savings: The Flan-T5 filter successfully pruned up to 60% of total tokens for product name tasks and 40% for software versions, leading to a proportional reduction in API costs.
  • Generalizability: The MCQ approach was also validated on the public FiNER-ORD financial dataset, demonstrating its effectiveness for identifying organization names in news articles beyond the tech support domain.

Conclusion and Future Work

This research presents a practical, cost-effective framework for automating product and software version tagging in enterprise support. By combining a lightweight filter classifier with MCQ-based LLM prompting, the system achieves high accuracy while significantly reducing operational costs.

Future research will focus on developing "dynamic ontologies" to automatically incorporate new products as they are released. Additionally, the team aims to address data privacy hurdles to facilitate the broader adoption of proprietary LLMs in sensitive enterprise environments.

Summary updated

Abstract

In a technology company, quality of customer service that involves providing troubleshooting assistance and advice to customers is a crucial asset.Often, insights from historical customer service data are used to make decisions related to future product offerings.In this paper, we address the challenging problem of automatic assignment of product names and software version labels to customer Service Requests (SRs) related to Cisco Systems, Inc., a global company specializing in networking hardware, software, and services. 1We study the effectiveness of state-of-the-art Large Language Models (LLMs) in assigning the correct product name codes and software versions from several possible label options and their "noncanonical" mentions in the associated SR data.We frame this assignment as a multiple-choice question (MCQ) answering task instead of conventional prompts and devise, to our knowledge, a novel pipeline of employing a classifier for filtering inputs to the LLM for saving usage costs.On our experimental dataset based on real SRs, we are able to correctly identify product name and software version labels when they are mentioned with over 90% accuracy while cutting LLM costs by about 40 -60% on average, thus providing a viable solution for practical deployment.
BibTeX
@misc{gollapalli2025assigning,
  author = {Sujatha Das Gollapalli and Mouad Hakam and Mingzhe Du and See-Kiong Ng and Mohammed Hamzeh},
  title = {{On Assigning Product and Software Codes to Customer Service Requests with Large Language Models}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-industry.76},
}
Figure