Performance Prediction for Conversational Search Using Perplexities of Query Rewrites

Chuan Meng, Mohammad Aliannejadi, Maarten de RijkeQPP++@ECIR 2023 · OpenReview

Abstract

Conversational search (CS) systems typically employ a two-stage pipeline: rewriting context-dependent user turns into standalone queries, followed by standard document retrieval. However, the quality of these rewrites is a major bottleneck; poor rewrites lead to irrelevant search results. This paper introduces PPL-QPP, a novel Query Performance Prediction (QPP) framework that utilizes the perplexity of query rewrites to estimate retrieval quality. By measuring the "naturalness" of a rewrite using a pre-trained language model, PPL-QPP identifies potentially problematic queries before retrieval. Experiments on the CAsT and OR-QuAC datasets show that PPL-QPP consistently outperforms traditional QPP methods, especially in scenarios where query rewriting is challenging.

Introduction

Conversational search is inherently complex due to the use of anaphora and ellipsis in human dialogue. To bridge the gap between conversational turns and ad-hoc search engines, models like T5 are used to generate "self-contained" rewrites. While Query Performance Prediction (QPP) is well-established for traditional search, its application in conversational contexts is limited. Existing QPP methods often fail because they do not account for the linguistic quality of the generated rewrite. If a rewriter produces nonsensical text, standard statistical QPP metrics (like term variance) may still provide misleadingly high scores.

The PPL-QPP Framework

The proposed Perplexity-based pre-retrieval QPP framework (PPL-QPP) introduces linguistic naturalness as a proxy for rewrite quality. The core hypothesis is that well-formed, natural language rewrites are more likely to yield successful search results.

PPL-QPP calculates the perplexity (PPL) of a T5-generated rewrite using a large-scale language model (GPT-2 XL). High perplexity indicates a poorly formed or nonsensical query. The final prediction score is derived through a linear interpolation of the inverse perplexity and a standard QPP metric:

$$\text{Score} = \alpha \cdot \frac{1}{PPL} + (1 - \alpha) \cdot QPP$$

where $\alpha$ is a tunable parameter that balances the linguistic signal with traditional statistical signals (e.g., VAR or SCQ).

Experimental Setup

The framework was evaluated using three major conversational search datasets:

  • CAsT-19 & CAsT-20: Benchmarks from the TREC Conversational Assistance Track.
  • OR-QuAC: An open-retrieval version of the Question Answering in Context dataset.

The baseline system utilized a T5-based rewriter and a BM25 retriever. Performance was measured using Pearson (P-$\rho$), Kendall (K-$\tau$), and Spearman (S-$\rho$) correlation coefficients between the predicted scores and actual retrieval performance (NDCG@3).

Results and Discussion

Rewriting Quality as a Performance Bottleneck

Analysis reveals a strong correlation between the quality of query rewrites and subsequent retrieval performance. As shown in Figure 1, datasets with lower ROUGE scores (indicating higher divergence from manual human rewrites) also exhibit lower NDCG@3 scores. This confirms that the rewriter's output quality is a critical factor in conversational search success.

Comparison of ROUGE scores and retrieval quality across datasets

PPL-QPP Performance

PPL-QPP demonstrated superior performance across most metrics, particularly on the CAsT-20 dataset, which presented the most significant rewriting challenges. By integrating perplexity, the framework effectively "sanity checks" the rewriter's output.

Performance of QPP methods across three datasets

As detailed in the table above, PPL-QPP achieved the highest correlation scores on CAsT-19 and CAsT-20. On CAsT-20, it improved the Pearson correlation from 0.210 (best baseline) to 0.231. On OR-QuAC, the framework matched the best baseline (VAR), suggesting that the perplexity signal is most beneficial when the rewriter is prone to generating low-quality text.

Conclusion

PPL-QPP addresses a vital gap in conversational search by incorporating linguistic naturalness into performance prediction. While the use of large language models for perplexity calculation introduces some computational overhead, the gains in prediction accuracy—especially for difficult queries—provide a robust mechanism for improving user experience in conversational systems. Future work will explore more efficient linguistic features and investigate the framework's performance on a wider variety of datasets.

Summary updated

BibTeX
@inproceedings{meng2023performance,
  author = {Chuan Meng and Mohammad Aliannejadi and Maarten de Rijke},
  title = {{Performance Prediction for Conversational Search Using Perplexities of Query Rewrites}},
  booktitle = {QPP++@ECIR},
  year = {2023},
  url = {https://openreview.net/forum?id=YjjG15pKBj_},
}
Figure