A multi-level approach for hierarchical Ticket Classification

Matteo Marcuzzo, Alessandro Zangari, Michele Schiavinato, Lorenzo Giudice, Andrea Gasparetto, Andrea AlbarelliCOLING 2022

TL;DR

This paper tackles the challenge of automatically classifying customer support tickets, which are often noisy and complex. The authors propose a novel multi-level classification approach using BERT, demonstrating that incorporating hierarchical label information significantly improves accuracy. Their work offers practical insights into leveraging pre-trained language models for this crucial business task.

What Problem Does This Paper Address?

Modern businesses receive a high volume of customer support tickets, often in the form of concise, sometimes messy, textual descriptions. Efficiently categorizing these tickets is crucial for quick issue resolution, customer satisfaction, and meeting service-level agreements. Traditional methods struggle with the inherent noise and jargon in these texts. While pre-trained language models like BERT have revolutionized text classification, their application to the specific domain of support tickets, especially with hierarchical labeling structures, remains an area ripe for exploration. This paper aims to bridge this gap by investigating how to best utilize BERT for hierarchical ticket classification, considering various document representation strategies and the impact of hierarchical label information.

Methods

The core of this work lies in adapting BERT, a powerful pre-trained language model, for hierarchical ticket classification. The authors explore several strategies for generating a single, representative embedding for each ticket from BERT's word embeddings. These include using the special [CLS] token (a common practice), averaging token embeddings, and concatenating embeddings from multiple layers, hypothesizing that different layers capture different linguistic features.

Beyond single-ticket representations, they introduce two novel multi-level classifier architectures:

  • ML-BERT: This approach uses two separate pre-trained BERT models, one fine-tuned for the top-level categories (macro-labels) and another for the sub-categories (flattened labels). The embeddings from these models are then concatenated and fed into a final linear layer.
  • Supported-BERT: Similar to ML-BERT, it uses a pre-trained model for macro-labels. However, the second model for sub-labels is fine-tuned with the first model providing support, aiming for a more integrated learning process.

Both multi-level approaches leverage the best-performing summarization strategy identified earlier and concatenate the resulting embeddings before a final classification layer.

Datasets

The primary dataset used for evaluation is the Linux Bugs dataset. This dataset consists of bug reports crawled from the publicly available Linux kernel bug tracker. It features a hierarchical structure with two levels: "product" (e.g., Network, Drivers) as the main labels and "component" (e.g., BIOS, scheduler) as sub-labels. The authors preprocessed this dataset by discarding reports without valid message text, lowercasing all text, and concatenating the issue title with the message body. They also filtered out labels and sub-labels appearing less than 100 times to mitigate class imbalance.

Key Insights and Ideas

  • Summarization Matters: The way a ticket's representation is summarized from BERT's token embeddings significantly impacts classification performance. Simply using the [CLS] token isn't always optimal; concatenating embeddings from multiple layers proved more effective.
  • Hierarchy is Key: Injecting hierarchical label information into the classification architecture provides a substantial boost in performance compared to treating all labels as flat. This suggests that the relationships between categories are valuable signals.
  • BERT's Robustness: BERT, despite its sensitivity to noise in some contexts, performs well on this noisy, user-generated support ticket data, especially when combined with effective summarization and hierarchical approaches.
  • Multi-level Architectures Pay Off: Both ML-BERT and Supported-BERT architectures demonstrate clear improvements over a standard BERT model trained on flattened labels, highlighting the benefit of explicitly modeling the hierarchy.

Key Results

The paper reports several key findings:

  • Summarization Strategy: The cls concat3 strategy (concatenating embeddings from the first three BERT layers) achieved the best overall performance in terms of accuracy and F1-score on the T2 (sub-label) task.
  • Multi-level Classifiers:
    • The ML-BERT (base) model achieved a 9.7% improvement in macro F1-score and a 5.4% accuracy improvement over a flattened BERT model.
  • The Supported-BERT (base) model showed even greater gains, with a 6.4% macro F1-score and 7.0% accuracy improvement.
  • While the larger BERT models showed a similar trend, their performance was slightly lower, possibly due to dataset size and complexity.
  • Comparison to SVM: A traditional SVM classifier with TF-IDF features performed surprisingly well, even outperforming the smaller BERT model in the flattened setting, suggesting that simpler methods can still be competitive on noisy text if they effectively filter out irrelevant information. However, the proposed multi-level BERT models still outperformed SVM.

Limitations and Future Work

  • Error Analysis: The authors acknowledge that a full in-depth error analysis was not performed due to time constraints. They plan to expand this in future work using specialized tools like Errudite, LIT, and iSEA to better understand model decisions and identify sources of errors, particularly for classes with low recall.
  • Dataset Size and Complexity: The performance of the larger BERT models was not as strong as expected, potentially due to the limited size and the presence of technical jargon (like stack traces) in the dataset, which might reduce the semantic significance of the text for the model.
  • Class Imbalance: While some steps were taken to address class imbalance, the paper notes that classes with fewer samples generally perform worse. Future work could explore advanced over- and under-sampling techniques to further improve performance on minority classes.
  • Semantic Similarity: The authors hypothesize that semantically similar labels (e.g., different types of "Drivers") might contribute to classification difficulty and plan to investigate this further.

Summary updated

BibTeX
@inproceedings{marcuzzo2022multi,
  author = {Matteo Marcuzzo and Alessandro Zangari and Michele Schiavinato and Lorenzo Giudice and Andrea Gasparetto and Andrea Albarelli},
  title = {{A multi-level approach for hierarchical Ticket Classification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure