Vertical Domain Text Classification: Towards Understanding IT Tickets Using Deep Neural Networks

Jianglei Han, Mohammad AkbariAAAI 2018 · DOI

Abstract

Automating the classification of IT support tickets is a high-stakes task where manual errors frequently lead to significant operational inefficiencies. This paper investigates the application of deep learning to this "vertical domain" problem, characterized by specialized technical language and high label cardinality. By comparing traditional rule-based and statistical methods against Convolutional Neural Networks (CNNs), the authors demonstrate that CNNs utilizing domain-specific pre-trained word vectors significantly outperform legacy systems, offering a robust solution for navigating the thousands of categories found in enterprise IT environments.

The IT Ticket Routing Problem

In large-scale IT support, every incident or change request begins as a "ticket" that must be routed to the correct technical team. Historically, this routing has been manual or guided by rigid, human-crafted rules. The authors identify three major pain points in this process:

  1. High Cardinality: Enterprise systems often have hundreds or thousands of potential categories, making it nearly impossible for human users to choose the correct one consistently.
  2. High Error Rates: In the system studied, nearly 40% of tickets required manual re-assignment because the initial category was incorrect.
  3. Brittleness: Rule-based systems are difficult to maintain and fail to scale as new technologies and problem types emerge.
Cumulative percentage of labels vs. number of tickets, illustrating the high cardinality and long-tail distribution of IT ticket categories.

Methodology

The researchers evaluated five distinct approaches to classify ticket titles:

  • Rule-based: A system utilizing over 1.4 million manually crafted lexical rules.
  • IR-based (Information Retrieval): A retrieval-based approach that matches new tickets against an index of historical examples.
  • SVM (Support Vector Machine): A traditional machine learning approach using TF-IDF features.
  • CNN-Rand: A Convolutional Neural Network where word embeddings are initialized randomly and learned during training.
  • CNN-Pretrain: A CNN using word vectors pre-trained on the specific IT dataset, allowing the model to capture domain-specific semantic relationships.

CNNs are particularly effective for text because they use convolutions to capture local semantic context (n-grams), preserving the local ordering information that is often lost in "bag-of-words" models like SVMs.

Dataset Analysis

The study utilized a real-world dataset from a major software enterprise, comprising 177,851 tickets created in 2016. The researchers focused exclusively on ticket titles, testing how much information could be extracted from these relatively short strings of text.

Distribution of ticket title lengths by word count (a) and character count (b), showing that most inputs are short, dense strings of text.

Key Results

The models were evaluated using "Accuracy@k," measuring if the correct label appeared in the model's top k predictions. The CNN-Pretrain model was the clear winner, achieving a Top-1 accuracy of 44.4% and a Top-10 accuracy of 76.1%.

Method Accuracy@1 Accuracy@3 Accuracy@5 Accuracy@10
Rule-based 0.037 0.072 0.098 0.140
SVM 0.232 0.341 0.386 0.460
IR-based 0.370 0.545 0.615 0.706
CNN-Pretrain 0.444 0.620 0.684 0.761

Key Insights

  • Domain-Specific Pre-training: Moving from random initialization to pre-trained word vectors yielded a nearly 15% improvement in Top-1 accuracy. This highlights that in specialized fields like IT, domain-specific jargon requires tailored language modeling.
  • The "Search" Alternative: The IR-based method performed surprisingly well, suggesting that for tasks with massive numbers of classes, looking for "similar past examples" is a robust baseline.
  • Rule-Based Failure: Despite having 1.4 million rules, the rule-based system performed the worst, underscoring the futility of trying to manually map the infinite variety of human language to rigid logic.

Limitations and Future Work

The current study only utilizes ticket titles. IT tickets typically contain longer descriptions and attachments which likely contain vital clues for classification. Future work should explore incorporating these richer data sources and testing more modern neural architectures, such as Transformers, to further improve accuracy in complex vertical domains.

Summary updated

BibTeX
@inproceedings{han2018vertical,
  author = {Jianglei Han and Mohammad Akbari},
  title = {{Vertical Domain Text Classification: Towards Understanding IT Tickets Using Deep Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11375},
}
Figure