Query Understanding in LLM-based Conversational Information Seeking

Yifei Yuan, Zahra Abbasiantaeb, Mohammad Aliannejadi, Yang Deng

Introduction

Traditional search engines have long focused on "ad-hoc retrieval," where users provide specific keywords to receive a list of relevant links. However, the paradigm is shifting toward Conversational Information Seeking (CIS). In CIS, interactions mimic human dialogue: queries are often vague, fragmented, or evolve over multiple turns. The core of this transition lies in Query Understanding—the system's capacity to interpret user intent accurately despite linguistic complexities like anaphora (pronoun usage), ellipsis (omitted context), and topic shifts.

Large Language Models (LLMs) have fundamentally altered this landscape. This paper synthesizes the current state-of-the-art, outlining how LLMs are moving query understanding beyond simple keyword matching toward proactive, context-aware interaction.

The Core Challenges of Conversational Understanding

Interpreting a single search query is straightforward; interpreting a ten-turn conversation is not. Three primary gaps persist in current CIS research:

  1. Evaluation Complexity: Measuring whether a system truly understands the nuances of a multi-turn dialogue remains difficult.
  2. Proactivity: Determining the optimal moment for a system to stop guessing and instead ask for clarification.
  3. Dynamic Intent: Managing users who refine their goals or change topics mid-stream.

A Four-Part Framework for LLM Integration

To address these challenges, researchers have developed a framework categorized into four key areas:

1. LLM-based Evaluation

Static benchmarks are increasingly insufficient for conversational AI. The field is moving toward using LLMs as "relevance assessors." In this role, a high-performing model (like GPT-4) evaluates the quality and relevance of another system's responses to complex, multi-turn queries. While efficient, this approach requires caution regarding "data leakage" and reproducibility.

2. User Simulation

Human trials are expensive and difficult to scale. By using LLMs to act as "synthetic users," researchers can simulate a vast array of user behaviors, intents, and personas. This allows for rigorous stress-testing of CIS systems under diverse conversational conditions without the overhead of human recruitment.

3. Proactive Management and Mixed-Initiative Systems

A significant evolution in CIS is the development of "mixed-initiative" systems. Instead of being purely reactive, these systems are trained to recognize when a query is too ambiguous to answer. They proactively ask clarifying questions or suggest follow-up paths, balancing helpfulness with the need to avoid user annoyance.

4. Query Enhancement and Rewriting

Techniques such as Conversational Query Rewriting use LLMs to transform brief, context-dependent chat messages into standalone search queries. This ensures that downstream retrieval components receive a clear, comprehensive input that captures the entire conversation's history.

Benchmarking Progress

The development of these systems relies on specialized datasets designed for conversational depth:

  • QReCC & TopioCQA: Large-scale datasets for open-domain conversational question answering.
  • TREC CAsT (2019-2022): A benchmark focused on complex, knowledge-intensive conversational tracks.
  • TREC iKAT (2023): A newer benchmark specifically designed for interactive knowledge assistants.
  • CLAMBER: A benchmark for identifying and clarifying ambiguous information needs.

Key Insights and the Future of CIS

The transition from passive response to active management represents a major shift in the field. However, this transition introduces the "Initiative Problem": current LLMs often excel at answering but struggle with strategic planning—knowing when to ask a question rather than providing a potentially incorrect answer.

Furthermore, the field is moving toward Multimodality. Future query understanding will likely incorporate images and audio, allowing users to seek information based on visual context (e.g., "Find me a shirt like the one in this photo, but in blue").

Conclusion and Open Challenges

While LLMs have revolutionized Natural Language Understanding in search—particularly in low-resource, zero-shot scenarios—several frontiers remain. Real-time adaptation to "intent shifts" and the creation of robust, dynamic evaluation metrics are critical. Additionally, bridging multilingual and cross-cultural gaps is essential to ensure that conversational systems can navigate the linguistic nuances of global users. The goal remains a "sweet spot" where system initiative enhances the user experience without becoming intrusive.

Summary updated

BibTeX
@inproceedings{yuan2025query,
  author = {Yifei Yuan and Zahra Abbasiantaeb and Mohammad Aliannejadi and Yang Deng},
  title = {{Query Understanding in LLM-based Conversational Information Seeking}},
  booktitle = {The Web Conference},
  year = {2025},
  url = {https://openreview.net/forum?id=QPH5JqJDUb},
}
Figure