Learning to Align Question and Answer Utterances in Customer Service Conversation with Recurrent Pointer Networks

Shizhu He, Kang Liu, Weiting AnAAAI 2019 · DOI

Abstract

In complex, multi-round customer service (CS) dialogues, identifying which agent response corresponds to which customer query is a significant challenge. This paper presents a novel approach using Recurrent Pointer Networks (RPN) to automate Utterance Alignment (UA). By leveraging role-specific encoding and a regression-based pointer mechanism, the proposed model effectively handles non-aligned "chitchat" and one-to-many response patterns, significantly outperforming existing word-matching and discriminative baselines.

The Utterance Alignment Problem

In ideal scenarios, customer service interactions follow a strict one-to-one question-and-answer format. However, real-world CS chats are often fragmented and asynchronous. A customer might send multiple messages to describe a single issue, or an agent might provide a single response that addresses several previous points. Furthermore, many utterances are purely social (greetings, thanks) and require no alignment.

Example of complex utterance alignments in customer service chats.

This task, known as Utterance Alignment (UA), is difficult for traditional models because:

  1. Contextual Dependency: Utterances like "Is it the same?" are ambiguous without historical context.
  2. Empty Alignments: A significant portion of messages (e.g., greetings) do not map to any specific query.
  3. One-to-Many Mapping: A single server response may resolve multiple distinct customer questions.

Methods: Recurrent Pointer Networks

The authors propose an end-to-end neural architecture centered on Pointer Networks. Unlike standard sequence-to-sequence models, Pointer Networks are designed to select specific indices from the input sequence, making them a natural fit for pointing from an answer back to its corresponding question(s).

Architectural overview of Model-I and Model-II.

The system architecture comprises three primary components:

  • Utterance Encoder: A Convolutional Neural Network (CNN) with role embeddings transforms individual messages into vectors while preserving the distinction between Customer and Server roles.
  • Conversation Encoder: Recurrent Neural Networks (RNNs) capture the temporal flow. The researchers evaluated two variants: Model-I (shared encoder) and Model-II (separate encoders for customer and server to reflect their differing knowledge domains).
  • Alignment Decoder: The core innovation uses a pointer mechanism to calculate the probability of alignment for every previous message. By employing a Regression Loss (via a sigmoid function) instead of standard classification, the model can independently assess each potential pair, naturally facilitating one-to-many alignments without requiring manual thresholds.

Experimental Setup and Datasets

The study utilized a new dataset derived from an in-house online customer service platform. After rigorous cleaning and manual labeling, the dataset included 5,741 conversations. Analysis revealed a challenging distribution: 57% of utterances had "None" (empty) alignments, and 12% involved "One-to-Many" alignments. On average, conversations consisted of approximately 10.5 total utterances.

Key Results and Insights

The RPN-based models demonstrated superior performance across all metrics compared to rule-based heuristics and previous state-of-the-art models (e.g., SHCNN+CISIR).

  • Overall Accuracy: The best-performing configuration, Model-II with Regression, achieved an overall F1-score of 84.86%, surpassing the previous state-of-the-art by over 5 percentage points.
  • Role-Specific Encoding: Model-II consistently outperformed Model-I, validating the hypothesis that customers and servers operate in distinct latent spaces (queries vs. solutions).
  • Robustness to Complexity: The model showed significant improvements in handling "None" alignments (94.07% F1) and the difficult "One-to-Many" category (80.39% F1).

Conclusion and Future Directions

Recurrent Pointer Networks provide a robust framework for structural alignment in messy dialogue data. While highly effective, the authors identify two paths for future research: integrating external product Knowledge Bases (KB) to improve semantic understanding (e.g., linking "fabric" to "silk") and exploring semi-supervised learning techniques to reduce the reliance on expensive manually labeled datasets.

Summary updated

BibTeX
@inproceedings{he2019learning,
  author = {Shizhu He and Kang Liu and Weiting An},
  title = {{Learning to Align Question and Answer Utterances in Customer Service Conversation with Recurrent Pointer Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.3301134},
}
Figure