Report on the 1st Workshop on Query Performance Prediction and Its Evaluation in New Tasks (QPP++ 2023) at ECIR 2023

Guglielmo Faggioli, Nicola Ferro, Josiane Mothe, Fiana Raiber, Maik FröbeSIGIR 2023 · OpenReview

TL;DR

This paper reports on the first QPP++ workshop, which addressed the growing need to adapt Query Performance Prediction (QPP) techniques to the evolving landscape of Information Retrieval (IR), particularly with the rise of new search paradigms like conversational search and the advancements in large language models. The workshop highlighted key challenges and proposed future directions for QPP research, emphasizing the need for more robust evaluation methods and better reproducibility.

What Problem Does This Paper Address?

Query Performance Prediction (QPP) is a crucial area in Information Retrieval (IR) that aims to estimate how well a search query will perform without requiring human relevance judgments. Traditionally, QPP has been applied to standard ad-hoc retrieval tasks. However, the IR field is rapidly advancing with new technologies like large language models (LLMs) and new search modalities such as conversational search, semantic search, and question answering. These advancements create a gap: existing QPP methods, often relying on traditional collection statistics, may not perform well in these new contexts. Furthermore, the evaluation of QPP itself needs re-evaluation to keep pace with these changes. The QPP++ 2023 workshop was convened to bridge this gap by fostering discussion on how to adapt QPP to new IR tasks and technologies, and how to effectively evaluate these new QPP approaches.

Methods

This paper is a report on a workshop, not a research paper presenting novel algorithms. The "methods" here refer to the workshop's structure and the discussions that took place. The workshop was divided into two parts:

  1. Presentation of Accepted Papers: Authors of accepted submissions presented their work, establishing a common ground and identifying key challenges.
  2. Focus Groups Discussion: A "world café" methodology was used, where participants (around 20) were split into small groups to discuss specific themes. The primary theme identified was the evaluation of QPP methods.

Datasets

Not discussed in the paper. The paper mentions that the proceedings of the QPP++ 2023 workshop are publicly available, implying that the accepted papers might detail specific datasets used in their respective research.

Key Insights and Ideas

  • The Need for QPP Adaptation: The rapid evolution of IR systems (LLMs, conversational search, semantic search) necessitates a fundamental rethinking of QPP. Traditional QPP methods, often based on collection statistics, may not generalize well to these new paradigms.
  • Challenges in Evaluating QPP: The standard method of evaluating QPP by correlating predicted scores with relevance judgments (e.g., Average Precision) has limitations. This correlation doesn't always translate to actual retrieval effectiveness, and its reliability is questionable across different IR models, corpora, and predictors. This issue is amplified in new domains like conversational search.
  • Beyond Correlation: Participants suggested that QPP evaluation should move beyond simple correlation and incorporate both intrinsic and extrinsic evaluations, including downstream tasks. This could involve tasks like selecting retrieval models, suggesting query reformulations, or identifying unanswerable queries.
  • Qualitative Analysis is Crucial: Current QPP evaluation often lacks qualitative analysis, focusing on overall performance rather than individual query performance. A deeper dive into specific queries is needed to identify weak spots. Scatterplots showing the relationship between predictions and actual performance were suggested as a way to achieve this.
  • Reproducibility is a Major Hurdle: The low reproducibility of QPP methods is a significant challenge. Even with the same ranking function, hyperparameter differences can lead to varied predictor behavior.
  • "Living Labs" for QPP: A novel idea proposed was the creation of "Living Labs" for QPP, where participants submit their runs, and these are shared for annotation and evaluation, potentially leading to leaderboard-style comparisons of QPP strategies.
  • Defining the User Model for QPP: While user satisfaction is well-defined in IR, the "user model" for QPP is less clear. Future research should focus on formally defining what it means to predict performance and clarifying the specific prediction target (e.g., query difficulty vs. actual performance).

Key Results

As a workshop report, this paper doesn't present quantitative experimental results in the traditional sense. Instead, it summarizes the consensus and key takeaways from the discussions:

  • Seven papers were accepted out of nine submissions, covering topics like QPP for conversational search, known-item search, learning-to-rank, and neural IR.
  • The primary challenge identified was QPP evaluation, leading to a dedicated focus group session.
  • Participants agreed on the need for improved evaluation methodologies that are more qualitative and consider downstream tasks.
  • A strong consensus emerged on the importance of releasing artifacts (code, ranked lists, models, experimental setup details) to improve reproducibility.

Limitations and Future Work

  • Limitations: The paper implicitly acknowledges the limitations of current QPP evaluation methods, which are often unreliable and lack qualitative depth. The challenge of applying QPP to new, complex IR tasks is also highlighted as an ongoing limitation.
  • Future Work:
    • Exploiting New Technologies: Investigate how LLMs, contextualized embeddings, and neural networks can be leveraged to improve QPP.
  • Adapting QPP to New Tasks: Develop and adapt QPP techniques for new domains like image retrieval, passage retrieval, question answering, and conversational search.
  • Improving Evaluation: Establish best practices, datasets, and resources for evaluating QPP, potentially developing new evaluation paradigms for different IR tasks.
  • Enhancing Reproducibility: Encourage the release of code, data, and models to facilitate replication and comparison of QPP approaches.
  • Formalizing QPP Objectives: Clearly define the prediction targets and theoretical underpinnings of QPP models.
  • "Living Labs": Explore the development and implementation of "Living Labs" for QPP to foster collaborative evaluation and benchmarking.

Summary updated

Abstract

Query Performance Prediction (QPP) is currently primarily applied to ad-hoc retrieval tasks. The Information Retrieval (IR) field is reaching new heights thanks to recent advances in large language models and neural networks, as well as emerging new ways of searching, such as conversational search. Such advancements are quickly spreading to adjacent research areas, including QPP, necessitating a reconsideration of how we perform and evaluate QPP. This workshop sought to elicit discussion on three topics related to the future of QPP: exploiting advances in IR to improve QPP, instantiating QPP on new search paradigms, and evaluating QPP on new tasks.Date: 6 April 2023.Website: https://qpp.dei.unipd.it/.
BibTeX
@inproceedings{faggioli2023report,
  author = {Guglielmo Faggioli and Nicola Ferro and Josiane Mothe and Fiana Raiber and Maik Fröbe},
  title = {{Report on the 1st Workshop on Query Performance Prediction and Its Evaluation in New Tasks (QPP++ 2023) at ECIR 2023}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  url = {https://openreview.net/forum?id=I9CzXORMYj},
}
Figure