Abstract
Conversational search requires models to maintain search intent across multiple dialogue turns, a task complicated by data scarcity and the inherent complexity of natural language. We present SSP (Self-Supervised Post-training), a novel training stage designed to bridge the gap between general-purpose ad-hoc retrievers and the specific requirements of conversational retrieval. By introducing three targeted self-supervised tasks—Topic Segmentation, Coreference Identification, and Word Reconstruction—SSP enables models to better capture dialogue structure and prevent "contextual semantic vanishing." Experimental results on TREC CAsT benchmarks demonstrate that SSP consistently improves state-of-the-art conversational search models, particularly in handling long-range dependencies and topic shifts.
Introduction
Unlike traditional ad-hoc search, conversational search involves user intents that are distributed across a sequence of utterances. A primary challenge is the resolution of coreferences and the maintenance of context; for instance, a model must recognize that a pronoun in the fifth turn refers to a subject introduced in the first.
Current progress in the field is often hindered by the scarcity of large-scale, high-quality conversational datasets. While many researchers employ knowledge distillation from pre-trained ad-hoc retrievers, these models frequently struggle with two issues: failing to recognize topic shifts (Dialogue Structure) and losing the core search intent as conversations lengthen (Contextual Semantic Vanishing).
Proposed Methodology
SSP is a post-training paradigm positioned between initial pre-training and final fine-tuning. It "warms up" the conversational encoder using three novel self-supervised tasks designed to model the nuances of dialogue.
1. Topic Segmentation Task
To improve the model's ability to identify topic boundaries, the training process injects "noise"—utterances from unrelated conversations—into the dialogue context. The model is then trained to distinguish between utterances belonging to the current topic and those that are irrelevant noise.
2. Coreference Identification Task
Leveraging existing query reformulation datasets, this task teaches the model to identify which specific previous utterance contains the term referred to by a pronoun or ellipsis in the current turn. This treats the dialogue as a linked structure rather than a flat sequence of text.
3. Word Reconstruction Task
To combat semantic vanishing, the model is required to reconstruct a Bag-of-Words (BoW) vector representing the entire conversation from its final vector representation. This forces the model to retain the global vocabulary and search intent throughout the dialogue.
Additionally, SSP utilizes Knowledge Distillation (KD) from a teacher model (a pre-trained ad-hoc retriever) to ensure the encoder maintains strong fundamental retrieval capabilities while learning these specialized conversational features.
Experimental Results
SSP was evaluated using the QReCC dataset for post-training and the TREC CAsT-19 and CAsT-20 benchmarks for evaluation.
Performance and Robustness
On the CAsT-20 dataset, the SSP-enhanced model (ConvDR-SSP) achieved a 7.1% improvement in MRR and a 6.7% improvement in NDCG@3 over previous state-of-the-art methods. Furthermore, "stress tests" involving the addition of random off-topic utterances showed that SSP-trained models are significantly more resilient to noise than standard encoders.
Ablation Study
Ablation analysis revealed that Coreference Identification is the most critical component of the framework. Removing this task resulted in the most substantial performance degradation, confirming that effective pronoun resolution is the cornerstone of conversational understanding.
Conclusion
SSP provides a versatile, "plug-and-play" strategy for enhancing conversational search models. By explicitly modeling dialogue structure and anchoring search intent through self-supervision, SSP overcomes the limitations of data scarcity and contextual decay. Future work will explore applying SSP to even more powerful base encoders and developing more complex structural modeling tasks.
Summary updated