SupportNet: Neural Networks for Summary Generation and Key Segment Extraction from Technical Support Tickets

Vinayshekhar Bannihatti Kumar, Mohan Yarramsetty, Sharon Sun, Anukul GoelACL 2021

TL;DR

This paper introduces SupportNet, a system designed to help technical support engineers resolve customer issues faster by providing them with automatically generated case summaries and relevant guidance snippets from past cases. By leveraging advanced NLP models like BERT and GPT-2, SupportNet aims to reduce the time engineers spend understanding complex technical problems, ultimately improving customer experience.

What Problem Does This Paper Address?

In the realm of technical support, especially for complex cloud services, customer issues can be deeply technical and time-sensitive. While previous efforts focused on reducing overall case resolution time through better routing and categorization, they often overlooked the significant portion of time engineers spend actively analyzing and understanding the customer's problem. This "active handle time" is crucial, as engineers need to sift through detailed correspondence, diagnose root causes, and formulate effective solutions. Scaling support operations to meet growing user demand necessitates improving the tools available to these engineers, particularly for handling business-critical and time-sensitive cases.

Methods

SupportNet comprises two main neural network-based components:

  1. Guidance Extraction Model: This system mines historical support cases to extract "technical guidance phrases." These phrases are intended to help engineers quickly find troubleshooting resources, understand root causes, and formulate responses. The model is built upon a pre-trained BERT model, further fine-tuned on a dataset of support cases. A key innovation here is the creation of "SupportBert," a BERT model that undergoes additional pre-training specifically on the domain of technical support tickets, aiming to better understand the jargon and nuances of this specific field.
  2. Summarization Model: This component generates an abstractive summary of a customer's problem from their case text (title and communication). It utilizes a Seq2Seq architecture, employing a BERT encoder to process the input text and a GPT-2 decoder to generate the summary. The model is trained using Maximum Likelihood Estimation (MLE) on customer case summaries. A notable aspect is the exploration of a multi-task learning approach, where the model is trained to simultaneously predict the issue category of the case, aiming to improve encoder representations.

Datasets

  • Guidance Extraction Dataset: This dataset consists of support case paragraphs labeled as either "Technical Guidance" (T.G.) or "Educational Guidance" (E.G.). The authors combined these into a single "guidance" category for classification. Statistics show a significant imbalance, with many more non-guidance examples than guidance examples.
  • Summarization Dataset: An internally available parallel corpus was used, containing customer case titles, the first communication from the customer, and the Subject Matter Expert (SME)-annotated customer problem. This dataset was used to train the encoder-decoder summarization model.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: The "SupportBert" model, which was further pre-trained on technical support ticket data, significantly outperformed an off-the-shelf BERT model for the guidance extraction task. This highlights the importance of tailoring language models to the specific domain they will operate in.
  • Multi-task Learning's Mixed Results: While multi-task learning (predicting case category alongside summarization) showed promise in improving the classifier's performance during early training, it surprisingly deteriorated the summarization metrics. The authors hypothesize this is due to conflicting gradient signals from the two tasks, forcing the encoder representations to adapt to both simultaneously.
  • Human Evaluation Matters: Automatic metrics like ROUGE and BERTScore are valuable, but human evaluation revealed that while summaries were often a good starting point, they sometimes struggled with semantic specificity, occasionally generating overly precise but contextually irrelevant details.

Key Results

  • Guidance Extraction: The SupportBert model (PB) achieved an F1 score of 0.64 on the guidance extraction task.
  • Summarization: The BERT encoder with GPT-2 decoder (BG) model performed best on ROUGE-L metrics, while the pre-trained BERT encoder with GPT-2 decoder (PBG) model slightly outperformed BG on BERTScore (F1) of 0.55.
  • Human Evaluation: For summarization, 40 out of 50 sampled cases were considered "useful" summaries by human evaluators if at least one expert found them useful.

Limitations and Future Work

  • Data Imbalance: The guidance extraction dataset is highly imbalanced, which can make achieving high performance challenging.
  • Multi-task Learning Degradation: The negative impact of multi-task learning on summarization performance needs further investigation.
  • Semantic Specificity in Summaries: The models sometimes generated summaries that were too specific and lacked broader semantic understanding, leading to penalties in human evaluation. Future work could focus on incorporating more semantic knowledge into the models.
  • Production Deployment: While promising, the authors suggest that further work is needed to deploy these models in production and assess their real-world impact on engineer efficiency and customer experience.

Summary updated

BibTeX
@inproceedings{kumar2021supportnet,
  author = {Vinayshekhar Bannihatti Kumar and Mohan Yarramsetty and Sharon Sun and Anukul Goel},
  title = {{SupportNet: Neural Networks for Summary Generation and Key Segment Extraction from Technical Support Tickets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure