Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retrieval in Asymmetric Texts

Pankaj Gupta, Bernt Andrassy, Hinrich Schütze

Abstract

The goal of our industrial ticketing system is to retrieve a relevant solution for an input query by matching it with historical tickets stored in a knowledge base. A query is comprised of a subject and description, while a historical ticket consists of a subject, description, and solution. To retrieve a relevant solution, we use a textual similarity paradigm to learn similarity between the query and historical tickets. The task is challenging due to significant term mismatch in query-ticket pairs of asymmetric lengths, where the subject is a short text but the description and solution are multi-sentence texts. We present a novel Replicated Siamese LSTM model to learn similarity in asymmetric text pairs, yielding a 22% and 7% gain in Accuracy@10 for the retrieval task over unsupervised and supervised baselines, respectively. We also demonstrate that topic and distributed semantic features for short and long texts improve both similarity learning and retrieval.

Introduction

In industrial environments, machine failures require engineers to search a knowledge base of historical tickets to find solutions. This process is essentially a Semantic Textual Similarity (STS) task, but it is complicated by two primary factors:

  1. Asymmetry: User queries are often short phrases (Subjects), whereas historical records contain short subjects, long paragraph-sized Descriptions, and multi-sentence Solutions. Matching a 5-word phrase to a 100-word paragraph is significantly more difficult than matching texts of equal length.
  2. Term Mismatch: Technical jargon often varies between users and manuals (e.g., "GT Trip" vs. "Narrow Frequency Pulsations"), causing traditional keyword-based search engines to fail.

Proposed Methodology

The core innovation is the Replicated Siamese LSTM architecture. Unlike standard Siamese networks that use two identical branches for one-to-one comparison, this model expands the architecture to handle multiple ticket components through cross-level learning and multi-channel features.

Overview of the Replicated Siamese LSTM architecture, including multi-channel features and cross-level learning components.

Multi-Level & Cross-Level Learning

The model learns to compare the Query Subject not just to the Ticket Subject, but also to the Ticket Description and Solution. This cross-level approach identifies semantic connections that simple one-to-one comparisons miss, as solutions often contain the specific keywords used in a query's description.

Multi-Channel Features

The similarity metric, a modified Manhattan distance, integrates three distinct feature channels:

  • LSTM Hidden Vectors ($h$): Captures sequential dependencies and local context.
  • Sum of Embeddings ($E$): A robust "bag-of-words" style average of word vectors (SumEMB).
  • Latent Topic Vectors ($T$): Derived from DocNADE (a neural topic model), these capture high-level themes like "vibration issues" or "fuel systems."

To handle specialized industrial vocabulary, the authors also incorporated a character-level BLSTM to understand the structure of rare technical codes and machine-specific jargon.

Experimental Results

The study utilized a real-world Industrial Ticketing Dataset from Siemens AG, consisting of 949 historical tickets and 421 manually labeled pairs. The Replicated Siamese LSTM (Model #22) significantly outperformed all baselines.

Performance comparison of various models across similarity and retrieval tasks.

Key performance metrics include:

  • Retrieval Accuracy: The model achieved a 22% gain in Accuracy@10 over unsupervised baselines (0.618 vs. 0.40) and a 7% gain over standard supervised Siamese LSTMs (0.618 vs. 0.548).
  • Error Reduction: The proposed model reduced the Mean Squared Error (MSE) in similarity prediction to 2.052, compared to 3.502 for the best unsupervised topic-based approach.
  • Qualitative Success: In end-user testing, the system reached a 77% success rate (providing at least one correct solution in the top 10) when considering both "Similar" and "Relevant" tickets.

Conclusion

The Replicated Siamese LSTM effectively addresses the challenges of asymmetric text matching in industrial ticketing systems. By combining sequential context with global topic features and cross-level comparisons, the model provides a significant boost in retrieval accuracy. Future work may involve incorporating attention mechanisms to focus on critical technical keywords and utilizing syntactic dependency trees to better model relationships between technical components.

Summary updated

Abstract

The goal of our industrial ticketing system is to retrieve a relevant solution for an input query, by matching with historical tickets stored in knowledge base. A query is comprised of subject and description, while a historical ticket consists of subject, description and solution. To retrieve a relevant solution, we use textual similarity paradigm to learn similarity in the query and historical tickets. The task is challenging due to significant term mismatch in the query and ticket pairs of asymmetric lengths, where subject is a short text but description and solution are multi-sentence texts. We present a novel Replicated Siamese LSTM model to learn similarity in asymmetric text pairs, that gives 22% and 7% gain (Accuracy@10) for retrieval task, respectively over unsupervised and supervised baselines. We also show that the topic and distributed semantic features for short and long texts improved both similarity learning and retrieval.
BibTeX
@inproceedings{gupta2018replicated,
  author = {Pankaj Gupta and Bernt Andrassy and Hinrich Schütze},
  title = {{Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retrieval in Asymmetric Texts}},
  booktitle = {SemDeep @ COLING},
  year = {2018},
  url = {https://openreview.net/forum?id=u3ClQY8u5mP},
}
Figure