SSP: Self-Supervised Post-training for Conversational Search

Quan Tu, Shen Gao, Xiaolong Wu, Zhao Cao, Ji-Rong Wen, Rui Yan

Abstract

Conversational search requires models to maintain search intent across multiple dialogue turns, a task complicated by data scarcity and the inherent complexity of natural language. We present SSP (Self-Supervised Post-training), a novel training stage designed to bridge the gap between general-purpose ad-hoc retrievers and the specific requirements of conversational retrieval. By introducing three targeted self-supervised tasks—Topic Segmentation, Coreference Identification, and Word Reconstruction—SSP enables models to better capture dialogue structure and prevent "contextual semantic vanishing." Experimental results on TREC CAsT benchmarks demonstrate that SSP consistently improves state-of-the-art conversational search models, particularly in handling long-range dependencies and topic shifts.

Introduction

Unlike traditional ad-hoc search, conversational search involves user intents that are distributed across a sequence of utterances. A primary challenge is the resolution of coreferences and the maintenance of context; for instance, a model must recognize that a pronoun in the fifth turn refers to a subject introduced in the first.

Comparison of standard conversational encoders versus SSP-enhanced encoders in tracking search intent across topic shifts.

Current progress in the field is often hindered by the scarcity of large-scale, high-quality conversational datasets. While many researchers employ knowledge distillation from pre-trained ad-hoc retrievers, these models frequently struggle with two issues: failing to recognize topic shifts (Dialogue Structure) and losing the core search intent as conversations lengthen (Contextual Semantic Vanishing).

Proposed Methodology

SSP is a post-training paradigm positioned between initial pre-training and final fine-tuning. It "warms up" the conversational encoder using three novel self-supervised tasks designed to model the nuances of dialogue.

The three self-supervised tasks of the SSP framework: Topic Segmentation, Word Reconstruction, and Coreference Identification.

1. Topic Segmentation Task

To improve the model's ability to identify topic boundaries, the training process injects "noise"—utterances from unrelated conversations—into the dialogue context. The model is then trained to distinguish between utterances belonging to the current topic and those that are irrelevant noise.

2. Coreference Identification Task

Leveraging existing query reformulation datasets, this task teaches the model to identify which specific previous utterance contains the term referred to by a pronoun or ellipsis in the current turn. This treats the dialogue as a linked structure rather than a flat sequence of text.

3. Word Reconstruction Task

To combat semantic vanishing, the model is required to reconstruct a Bag-of-Words (BoW) vector representing the entire conversation from its final vector representation. This forces the model to retain the global vocabulary and search intent throughout the dialogue.

Additionally, SSP utilizes Knowledge Distillation (KD) from a teacher model (a pre-trained ad-hoc retriever) to ensure the encoder maintains strong fundamental retrieval capabilities while learning these specialized conversational features.

Experimental Results

SSP was evaluated using the QReCC dataset for post-training and the TREC CAsT-19 and CAsT-20 benchmarks for evaluation.

Performance and Robustness

On the CAsT-20 dataset, the SSP-enhanced model (ConvDR-SSP) achieved a 7.1% improvement in MRR and a 6.7% improvement in NDCG@3 over previous state-of-the-art methods. Furthermore, "stress tests" involving the addition of random off-topic utterances showed that SSP-trained models are significantly more resilient to noise than standard encoders.

Ablation Study

Ablation analysis revealed that Coreference Identification is the most critical component of the framework. Removing this task resulted in the most substantial performance degradation, confirming that effective pronoun resolution is the cornerstone of conversational understanding.

Conclusion

SSP provides a versatile, "plug-and-play" strategy for enhancing conversational search models. By explicitly modeling dialogue structure and anchoring search intent through self-supervision, SSP overcomes the limitations of data scarcity and contextual decay. Future work will explore applying SSP to even more powerful base encoders and developing more complex structural modeling tasks.

Summary updated

Abstract

Conversational search has been regarded as the next-generation search paradigm. Constrained by data scarcity, most existing methods distill the well-trained ad-hoc retriever to the conversational retriever. However, these methods, which usually initialize parameters by query reformulation to discover contextualized dependency, have trouble in understanding the dialogue structure information and struggle with contextual semantic vanishing. In this paper, we propose {pasted macro ‘FULLMODEL’} ({pasted macro ‘MODEL’}) which is a new post-training paradigm with three self-supervised tasks to efficiently initialize the conversational search model to enhance the dialogue structure and contextual semantic understanding. Furthermore, the {pasted macro ‘MODEL’} can be plugged into most of the existing conversational models to boost their performance. To verify the effectiveness of our proposed method, we apply the conversational encoder post-trained by {pasted macro ‘MODEL’} on the conversational search task using two benchmark datasets: CAsT-19 and CAsT-20.Extensive experiments that our {pasted macro ‘MODEL’} can boost the performance of several existing conversational search methods. Our source code is available at https://github.com/morecry/SSP.
BibTeX
@inproceedings{tu2023ssp,
  author = {Quan Tu and Shen Gao and Xiaolong Wu and Zhao Cao and Ji-Rong Wen and Rui Yan},
  title = {{SSP: Self-Supervised Post-training for Conversational Search}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.837},
}
Figure