Abstract
Formulating effective search queries remains a challenging task, particularly when users lack expertise in a specific domain or are not proficient in the language of the content. Providing example documents of interest might be easier for a user. However, such query-by-example scenarios are prone to concept drift, and the retrieval effectiveness is highly sensitive to the query generation method, without a clear way to incorporate user feedback. To enable exploration and to support Human-In-The-Loop experiments we propose QueryExplorer -- an interactive query generation, reformulation, and retrieval interface with support for HuggingFace generation models and PyTerrier's retrieval pipelines and datasets, and extensive logging of human feedback. To allow users to create and modify effective queries, our demo supports complementary approaches of using LLMs interactively, assisting the user with edits and feedback at multiple stages of the query formulation process. With support for recording fine-grained interactions and user annotations, QueryExplorer can serve as a valuable experimental and research platform for annotation, qualitative evaluation, and conducting Human-in-the-Loop (HITL) experiments for complex search tasks where users struggle to formulate queries.
Introduction
In the realm of information retrieval, the "vocabulary gap" remains a significant barrier to effective search. Users often struggle to find relevant information when they lack domain-specific jargon or are searching across languages they do not speak. While "Query-by-Example" (QBE) approaches—where a user provides a relevant document to find similar content—offer a potential solution, they are frequently implemented as "black box" systems. These systems often suffer from concept drift and provide users with little to no control over how their intent is translated into a search query.
QueryExplorer is an open-source, interactive interface designed to bridge this gap. By combining the generative power of Large Language Models (LLMs) with robust retrieval pipelines, it transforms the search process into a collaborative, human-in-the-loop (HITL) experience. It serves both as a functional tool for complex search tasks and as a laboratory for researchers to study human-AI interaction in retrieval contexts.
System Architecture and Methods
QueryExplorer is built using the Gradio web framework, integrating LLMs from HuggingFace with retrieval pipelines from PyTerrier. The system's architecture is centered around three functional pillars:
1. Query Generation (QG)
To initiate a search from an example document, the system utilizes flan-t5-xxl. It employs a "few-shot" prompting strategy, using exemplars from the MSMarco dataset to guide the model in generating high-quality, representative search queries from the provided text. Unlike traditional QBE systems, the generated query is presented to the user for inspection and manual editing.
2. Query Reformulation (QR)
If initial results are too broad or misaligned, QueryExplorer offers automated reformulation. Using zero-shot prompting, the system suggests expansion terms and keywords. This allows users to refine their search strategy without needing to master the underlying domain vocabulary themselves.
3. Interactive Retrieval and Relevance Feedback
The system facilitates a dynamic feedback loop. Users can review retrieved documents and provide "Relevance Feedback" by selecting specific results and requesting the system to "find more like this." This action triggers the LLM to regenerate or adjust keywords based on the specific context of the selected document, effectively steering the search in real-time.
A Platform for Research
Beyond its utility as a search assistant, QueryExplorer is designed as a robust experimental platform. A dedicated Researcher’s Tab allows specialists to modify the underlying system components on the fly, including:
- Model Selection: Swapping different HuggingFace models for generation.
- Prompt Engineering: Adjusting the few-shot or zero-shot prompts used for QG and QR.
- Retrieval Algorithms: Switching between traditional methods like BM25 and modern dense retrieval techniques.
Crucially, the tool features fine-grained logging. Every user interaction—including query edits, slider adjustments for relevance, and reformulation choices—is captured in JSON format. This instrumentation allows researchers to perform qualitative evaluations and analyze the evolution of search strategies during complex tasks.
Datasets and Integration
QueryExplorer leverages several key IR resources to ensure broad applicability:
- MSMarco: Provides the foundation for few-shot query generation.
- IR-Datasets: Integrated via PyTerrier, allowing the tool to be deployed across a wide variety of standard benchmarks.
- BETTER Search Tasks: The system was originally developed to address the complexities of multi-stage search tasks within the BETTER program.
Discussion and Future Work
QueryExplorer demonstrates the feasibility of a modular, interactive approach to AI-assisted search. By providing transparency in the query generation process and allowing for human intervention, it mitigates the risks of concept drift inherent in automated QBE systems. Furthermore, its support for cross-lingual search—displaying translations alongside results—makes it a powerful tool for global information exploration.
However, the use of LLMs introduces challenges, such as the potential for generating biased or toxic keywords, which necessitates the use of safety filters. While the current Python-based implementation is optimized for research environments, future work will focus on scaling these interactive pipelines for high-traffic production environments and conducting large-scale user studies to quantify the impact of HITL query generation on search success.
Summary updated