Abstract
Automating the classification of IT support tickets is a high-stakes task where manual errors frequently lead to significant operational inefficiencies. This paper investigates the application of deep learning to this "vertical domain" problem, characterized by specialized technical language and high label cardinality. By comparing traditional rule-based and statistical methods against Convolutional Neural Networks (CNNs), the authors demonstrate that CNNs utilizing domain-specific pre-trained word vectors significantly outperform legacy systems, offering a robust solution for navigating the thousands of categories found in enterprise IT environments.
The IT Ticket Routing Problem
In large-scale IT support, every incident or change request begins as a "ticket" that must be routed to the correct technical team. Historically, this routing has been manual or guided by rigid, human-crafted rules. The authors identify three major pain points in this process:
- High Cardinality: Enterprise systems often have hundreds or thousands of potential categories, making it nearly impossible for human users to choose the correct one consistently.
- High Error Rates: In the system studied, nearly 40% of tickets required manual re-assignment because the initial category was incorrect.
- Brittleness: Rule-based systems are difficult to maintain and fail to scale as new technologies and problem types emerge.
Methodology
The researchers evaluated five distinct approaches to classify ticket titles:
- Rule-based: A system utilizing over 1.4 million manually crafted lexical rules.
- IR-based (Information Retrieval): A retrieval-based approach that matches new tickets against an index of historical examples.
- SVM (Support Vector Machine): A traditional machine learning approach using TF-IDF features.
- CNN-Rand: A Convolutional Neural Network where word embeddings are initialized randomly and learned during training.
- CNN-Pretrain: A CNN using word vectors pre-trained on the specific IT dataset, allowing the model to capture domain-specific semantic relationships.
CNNs are particularly effective for text because they use convolutions to capture local semantic context (n-grams), preserving the local ordering information that is often lost in "bag-of-words" models like SVMs.
Dataset Analysis
The study utilized a real-world dataset from a major software enterprise, comprising 177,851 tickets created in 2016. The researchers focused exclusively on ticket titles, testing how much information could be extracted from these relatively short strings of text.
Key Results
The models were evaluated using "Accuracy@k," measuring if the correct label appeared in the model's top k predictions. The CNN-Pretrain model was the clear winner, achieving a Top-1 accuracy of 44.4% and a Top-10 accuracy of 76.1%.
| Method | Accuracy@1 | Accuracy@3 | Accuracy@5 | Accuracy@10 |
|---|---|---|---|---|
| Rule-based | 0.037 | 0.072 | 0.098 | 0.140 |
| SVM | 0.232 | 0.341 | 0.386 | 0.460 |
| IR-based | 0.370 | 0.545 | 0.615 | 0.706 |
| CNN-Pretrain | 0.444 | 0.620 | 0.684 | 0.761 |
Key Insights
- Domain-Specific Pre-training: Moving from random initialization to pre-trained word vectors yielded a nearly 15% improvement in Top-1 accuracy. This highlights that in specialized fields like IT, domain-specific jargon requires tailored language modeling.
- The "Search" Alternative: The IR-based method performed surprisingly well, suggesting that for tasks with massive numbers of classes, looking for "similar past examples" is a robust baseline.
- Rule-Based Failure: Despite having 1.4 million rules, the rule-based system performed the worst, underscoring the futility of trying to manually map the infinite variety of human language to rigid logic.
Limitations and Future Work
The current study only utilizes ticket titles. IT tickets typically contain longer descriptions and attachments which likely contain vital clues for classification. Future work should explore incorporating these richer data sources and testing more modern neural architectures, such as Transformers, to further improve accuracy in complex vertical domains.
Summary updated