Q2R: A Query-to-Resolution System for Natural-Language Queries

Shiau Hong Lim, Laura WynterNAACL 2022 · DOI

Abstract

Traditional document retrieval systems often struggle in technical domains where queries and resolution documents exhibit a significant "vocabulary gap." We present Q2R (Query-to-Resolution), a hybrid retrieval system that reframes document search as a massive multiclass classification problem. By integrating a kernel-based K-Nearest Neighbor (KNN) approach—which leverages historical query-resolution pairs—with traditional content-based search, Q2R achieves superior performance in real-world technical support scenarios.

Introduction

In specialized fields such as IT support or medicine, users often describe problems in natural language that shares little to no overlap with the technical language used in solution documents. For instance, a user might report that their "laptop screen is flickering," while the corresponding fix is titled "Update BIOS version 2.1." This discrepancy, known as the vocabulary gap, poses a major challenge for standard retrieval models like BM25 or BERT-based rankers that rely on semantic or keyword matching.

Q2R addresses this by treating documents as unique classes. Instead of matching a query directly to a document's content, the system matches the new query to a database of historical queries that were successfully resolved by specific documents. This approach allows the system to learn the underlying intent behind a query, bypassing the limitations of document text.

Methodology

The Q2R architecture is composed of three synergistic components:

1. Kernel KNN (Direct Classification)

The core of Q2R is a nonparametric classification module. When a new query arrives, the system compares it against a library of past queries. If the new query is found to be similar to several historical queries that all pointed to a specific document, that document receives a high relevance score. This method is particularly effective because it can scale to a growing number of documents without requiring the entire model to be retrained.

2. Metric Learning via Triplet Loss

To ensure the KNN module is effective, the system must map queries into a meaningful embedding space. The authors employ metric learning using a Triplet Loss objective. During training, the model is presented with "triplets" consisting of an anchor query, a positive example (a query resolved by the same document), and a negative example (a query resolved by a different document). The goal is to minimize the distance between the anchor and the positive example while maximizing the distance to the negative example. An iterative sampling strategy is used to identify the most informative triplets, accelerating convergence.

3. The Q2R Orchestrator

While the KNN approach is powerful, it suffers from a "cold start" problem: it cannot retrieve documents that have no historical query data. To mitigate this, Q2R includes a standard content-based retriever as a fallback. The Q2R Orchestrator, a logistic regression classifier, analyzes the scores and confidence levels from both the KNN and content-based systems to determine the most reliable prediction for any given query.

Experimental Setup

The system was evaluated across three distinct datasets to test its versatility and scalability:

  • Twitter: A public dataset comprising 10,587 labeled dialog-to-URL pairs.
  • Telco: A domain-specific dataset with 38,687 examples from the telecommunications sector.
  • IBM: A large-scale industrial dataset containing over 433,000 training examples and nearly 150,000 unique URLs, reflecting the complexity of real-world technical support.

Results and Discussion

The experimental results demonstrate that Q2R consistently outperforms baseline models. On the Twitter dataset, the full Q2R system achieved a Mean Reciprocal Rank (MRR) of 0.557, a significant improvement over vanilla BM25 (0.079) and even a fine-tuned SBERT model (0.546).

Table 2: Performance comparison on the Twitter dataset. The Q2R Orchestrator (A+C and B+C) achieves the highest MRR and Recall scores.

A key insight from the study is the relationship between document "coverage" (the number of historical queries associated with a document) and model performance. As shown in the coverage analysis, the KNN-based models begin to dominate once a document has been seen at least 10 times in the training data. For documents with very high coverage (339+ examples), the performance gap between KNN and traditional content-based methods becomes even more pronounced.

Table 5: Performance by training coverage on the Twitter dataset. Content-based methods are essential for low-coverage documents, while Kernel KNN excels as historical data increases.

On the massive IBM dataset, the system proved its utility for large-scale industrial applications, with the KNN components significantly outperforming content-based approaches in overall average performance.

Conclusion and Future Work

Q2R demonstrates that historical resolution data is a potent resource for overcoming the vocabulary gap in technical document retrieval. By treating documents as classes and leveraging metric learning, the system provides a robust solution for high-traffic support environments.

However, the system's reliance on historical data remains its primary limitation. Future work will focus on improving the "cold start" performance for new documents and exploring more advanced fusion techniques for the Orchestrator. Q2R is currently in active use at IBM, proving the practical viability of this hybrid classification-retrieval approach.

Summary updated

BibTeX
@inproceedings{lim2022q,
  author = {Shiau Hong Lim and Laura Wynter},
  title = {{Q2R: A Query-to-Resolution System for Natural-Language Queries}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-industry.39},
}
Figure