Contextual Rephrase Detection for Reducing Friction in Dialogue Systems

Zhuoyi Wang, Saurabh Gupta, Jie Hao, Xing Fan, Dingcheng Li, Alexander Hanbo Li, Chenlei Guo

Zhuoyi Wang, Saurabh Gupta, Jie Hao, Xing Fan, Dingcheng Li, Alexander Hanbo Li, Chenlei Guo. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Abstract

Researchers at Amazon Alexa AI and UT Dallas have developed ContReph, a model that identifies when a user rephrases a query to correct a voice assistant's mistake. By moving beyond simple pairwise text similarity and incorporating full dialogue context—including agent responses and the precise timing between turns—ContReph significantly outperforms state-of-the-art baselines in reducing "friction" in conversational AI.

The Challenge of Dialogue Friction

When a voice assistant misinterprets a command—for example, hearing "play jack in the fall" when the user said "play jacking the ball"—the resulting user experience is defined as friction. Users typically respond by rephrasing their request until the system succeeds. Detecting these rephrases is essential for building "self-learning" systems that can automatically identify and rectify their own errors.

Historically, this has been treated as a pairwise similarity task, where the system compares two consecutive queries to determine if they are semantically identical. However, this approach is limited; it ignores the temporal context and the agent's own feedback. A user might repeat a similar query much later in a different context, or they might provide explicit corrective feedback that a simple similarity model would overlook. The goal of this research is to leverage the entire multi-turn dialogue to accurately pinpoint which previous query is being corrected.

Comparison of contextual rephrase detection vs. traditional pairwise detection.

Methodology: The ContReph Architecture

The authors propose ContReph, a contextual rephrase detection model built on the BERT architecture. The model introduces several key innovations to move beyond pairwise comparisons:

  • Dialogue Flattening: The entire dialogue session is treated as a single sequence, using special [USER] and [AGENT] tokens to maintain participant distinction.
  • Span Prediction: Instead of a binary classification, the model treats rephrase detection as a span prediction problem. Given a failed "source" query, the model predicts the start and end indices of the successful rephrase within the dialogue history.
  • Time-Bin Token Embeddings: Recognizing that users rephrase quickly when frustrated, the authors map the time difference between turns into discrete "bins" (e.g., intervals within a [-60, 60] second range). These are added as embeddings to the input layer to provide a temporal signal.
  • Implicit Feedback Integration: By including agent responses (e.g., "Sorry, I didn't find that"), the model learns to recognize system failures, which increases the likelihood that a subsequent similar query is a rephrase.
The ContReph architecture utilizing BERT with time-bin and token embeddings for span prediction.

Datasets and Evaluation

The study utilizes two primary datasets derived from large-scale, anonymous historic interactions:

  1. Machine-Annotated Set: A dataset of over 600,000 sessions labeled using an existing high-confidence Absorbing Markov Chain model.
  2. Human-Annotated Set: A "gold standard" set of 5,463 sessions where automated labeling failed, but human experts identified clear rephrases, representing a more difficult distribution of real-world friction.

Key Results

ContReph demonstrated significant improvements over state-of-the-art pairwise models like SimCSE and BERT-NSP:

  • Superior Accuracy: On the Human Test Set, ContReph achieved a 57.45% Exact Match (EM) score, outperforming the best pairwise baseline (BERT-NSP) by 27.89%.
  • Impact of Temporal Signals: Removing the time-bin encoding led to a drop of approximately 1.4% to 1.5% in EM scores, confirming that timing is a vital, albeit small, component of rephrase detection.
  • Robustness: While pairwise models struggled with the complex "Human-Annotated" set, ContReph remained robust, suggesting better generalization to real-world conversational nuances.

Discussion and Future Work

While ContReph marks a significant step forward, the authors noted that performance tends to degrade as dialogue length increases due to "long dialogue noise," where unrelated earlier turns can confuse the model's self-attention mechanism. Additionally, the model occasionally produces false triggers for distinct but similar requests made in quick succession.

Future research will explore contrastive learning to further improve query representations and focus on deploying the model in live production environments to measure direct improvements in user experience.

Summary updated

BibTeX
@inproceedings{wang2021contextual,
  author = {Zhuoyi Wang and Saurabh Gupta and Jie Hao and Xing Fan and Dingcheng Li and Alexander Hanbo Li and Chenlei Guo},
  title = {{Contextual Rephrase Detection for Reducing Friction in Dialogue Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=bHEplLZpW1Q},
}
Figure