SSP: Self-Supervised Post-training for Conversational Search

Quan Tu, Shen Gao, Xiaolong Wu, Zhao Cao, Ji-Rong Wen, Rui Yan

Abstract

Conversational search has been regarded as the next-generation search paradigm. Constrained by data scarcity, most existing methods distill the well-trained ad-hoc retriever to the conversational retriever. However, these methods, which usually initialize parameters by query reformulation to discover contextualized dependency, have trouble in understanding the dialogue structure information and struggle with contextual semantic vanishing. In this paper, we propose Self-Supervised Post-training (SSP) which is a new post-training paradigm with three self-supervised tasks to efficiently initialize the conversational search model to enhance the dialogue structure and contextual semantic understanding. Furthermore, the SSP can be plugged into most of the existing conversational models to boost their performance. To verify the effectiveness of our proposed method, we apply the conversational encoder post-trained by SSP on the conversational search task using two benchmark datasets: CAsT-19 and CAsT-20. Extensive experiments show that our SSP can boost the performance of several existing conversational search methods.

1. Introduction

Traditional "ad-hoc" search engines are designed for isolated, keyword-heavy queries. In contrast, conversational search involves fluid, multi-turn interactions where users rely on context, pronouns, and implicit references. This paradigm shift introduces two major technical hurdles: Structural Blindness and Contextual Semantic Vanishing.

Structural blindness refers to a model's inability to recognize when a user shifts topics—for example, moving from a discussion about throat cancer to lung cancer. Contextual semantic vanishing occurs as conversations lengthen; the mathematical representation of early turns often fades, causing the model to lose the user's original intent. Current models often attempt to solve this by distilling knowledge from ad-hoc retrievers, but they frequently fail to capture the underlying dialogue structure.

Overview of the SSP framework. The top-left shows how standard encoders fail to resolve coreferences or detect topic shifts, while the SSP-enhanced encoder (right) correctly identifies the search intent.

2. Methodology: The SSP Framework

We propose SSP, a post-training stage that bridges the gap between general pre-training and task-specific fine-tuning. SSP "warms up" the conversational encoder using three novel self-supervised tasks that do not require expensive manual labeling for every new domain.

2.1 Topic Segmentation

To help the model identify topic boundaries, we inject "noise" utterances from unrelated conversations into the current context. As shown in the architecture, the model must predict which utterances belong to the original conversation and which are intruders. This teaches the encoder to be sensitive to semantic shifts between turns.

2.2 Word Reconstruction

To prevent semantic vanishing, we force the model to reconstruct a Bag-of-Words (BoW) representation of the entire conversation history from its final vector representation. This ensures that the encoder retains critical information from early turns throughout the entire encoding process.

2.3 Coreference Identification

This task addresses the "it" problem. Using query reformulation data as a ground truth, the model is trained to identify exactly which previous turn contains the entity being referred to in the current query. By locating the source of a coreference, the model can effectively "rewrite" the query internally to include the missing context.

The internal architecture of SSP's three tasks. Topic Segmentation uses a shared encoder to classify utterances; Word Reconstruction uses an MLP to recover the context BoW; Coreference Identification locates the relevant previous turn.

3. Experimental Evaluation

We utilized the QReCC dataset for the self-supervised post-training phase, which contains 14K conversations with manual query reformulations. For evaluation, we tested on the CAsT-19 and CAsT-20 benchmarks, the gold standards for conversational assistance.

3.1 Key Results

The SSP framework consistently outperformed strong baselines across multiple metrics. On CAsT-19, the SSP-enhanced model (ConvDR-SSP) achieved an MRR of 0.780, a significant improvement over the standard ConvDR's 0.740. On the more challenging CAsT-20 dataset, performance improved from 0.501 to 0.526 MRR.

3.2 Robustness and Ablation

Ablation studies confirmed that Coreference Identification is the most critical component; its removal caused the largest performance drop. Furthermore, the Topic Segmentation task made the model significantly more robust to noise. In tests where "off-topic" noise was intentionally added, the SSP-enhanced model maintained stable performance while standard models saw their accuracy plummet.

4. Conclusion

SSP provides a versatile "plug-and-play" booster for conversational search. By focusing on the structural and semantic nuances of dialogue through self-supervision, it effectively addresses the limitations of current distillation-based methods. Future work will focus on adapting SSP to even stronger encoders and exploring more complex tasks to model deeper conversational dynamics.

Summary updated

Abstract

Conversational search has been regarded as the next-generation search paradigm. Constrained by data scarcity, most existing methods distill the well-trained ad-hoc retriever to the conversational retriever. However, these methods, which usually initialize parameters by query reformulation to discover contextualized dependency, have trouble in understanding the dialogue structure information and struggle with contextual semantic vanishing. In this paper, we propose \fullmodel (\model) which is a new post-training paradigm with three self-supervised tasks to efficiently initialize the conversational search model to enhance the dialogue structure and contextual semantic understanding. Furthermore, the \model can be plugged into most of the existing conversational models to boost their performance. To verify the effectiveness of our proposed method, we apply the conversational encoder post-trained by \model on the conversational search task using two benchmark datasets: CAsT-19 and CAsT-20. Extensive experiments that our \model can boost the performance of several existing conversational search methods. Our source code is available at \url{https://github.com/morecry/SSP}.
BibTeX
@inproceedings{tu2023ssp,
  author = {Quan Tu and Shen Gao and Xiaolong Wu and Zhao Cao and Ji-Rong Wen and Rui Yan},
  title = {{SSP: Self-Supervised Post-training for Conversational Search}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.837},
}
Figure