Introduction
In natural human dialogue, efficiency is paramount. We frequently omit words (ellipsis) or use pronouns (co-reference) when the context is clear. However, for conversational AI, these linguistic shortcuts pose a significant challenge. A query like "I can't find it" is ambiguous without the preceding context. Conversational Query Rewriting (CQR) aims to transform such incomplete fragments into self-contained, grammatically correct sentences that can be processed by downstream systems.
The Challenges of CQR
Despite its importance, CQR faces three primary hurdles:
- Data Scarcity: Training robust neural models requires vast amounts of manually rewritten query pairs, which are expensive and time-consuming to annotate.
- Keyword Neglect: Many models prioritize nouns (entities) but often fail to capture verbs or adjectives that are critical to the query's meaning.
- Intent Drift: Without explicit constraints, a rewritten query might become grammatically sound but lose the original user's intent.
The Teresa Framework
To address these issues, this paper introduces Teresa (Transformer-based query rewriting model), a novel framework that leverages self-supervised learning and structural constraints to improve rewrite quality.
1. Self-Supervised Learning (SSL) for Data Generation
Teresa overcomes data scarcity by automatically generating training pairs from raw dialogue logs. By "corrupting" existing full sentences—either by deleting words found in the context (simulating ellipsis) or replacing them with pronouns (simulating co-reference)—the authors created the JDDC-CQR-10M dataset. This massive synthetic corpus allows the model to learn the mechanics of rewriting before fine-tuning on high-quality manual data.
2. Self-Attentive Keywords Detection (SAKD)
To prevent keyword neglect, Teresa employs a graph-based approach. It constructs a graph using the encoder's self-attention weights and applies the TextRank algorithm to identify salient words, including crucial verbs and adjectives. These scores guide the model’s copy mechanism, ensuring that vital semantic components are preserved in the final rewrite.
3. Intent Consistency Constraint (ICC)
To prevent intent drift, Teresa uses a shared encoder to represent the intent of both the original contextual query and the rewritten version. By applying a Kullback-Leibler (KL) divergence loss, the model is forced to maintain the same underlying meaning across both representations, acting as a mathematical anchor against "hallucinations."
Experimental Results
The researchers evaluated Teresa against several state-of-the-art baselines on benchmarks including JDDC-CQR-146K and Restoration-200K.
- Performance: Teresa achieved a BLEU-4 score of 79.62 on the JDDC dataset, significantly outperforming the best baseline (73.34).
- Exact Match (EM): For queries requiring rewriting (positive samples), Teresa reached an EM of 50.82%, a substantial improvement over previous models (32.40%).
- Efficiency of SSL: Pre-training on synthetic data proved highly effective. Even with only 10% of the manually annotated data, the SSL-pre-trained model showed remarkable performance, demonstrating the power of the automated data generation approach.
Conclusion
Teresa represents a significant step forward in Conversational Query Rewriting. By combining self-supervised data generation with keyword detection and intent consistency constraints, it produces more accurate and contextually faithful rewrites. Future work may involve integrating these CQR capabilities directly into the pre-training of large-scale language models to create more universal and robust conversational agents.
Summary updated