Diversifying Information Needs in Results of Question Retrieval

Yaoyun Zhang, Xiaolong Wang, Xuan Wang, Ruifeng Xu, Jun Xu, Shixi Fan

Introduction

Standard question retrieval systems often return a list of semantically similar questions that satisfy the same narrow intent. However, a user's actual "information need" (infoNeed) depends heavily on context. For example, a query like "Which bank provides the best credit card?" could imply a need for a specific name (Entity), a comparison of rates (Comparison), or a guide on how to apply (Method). This paper introduces a method to diversify search results by leveraging CogQTaxo, a three-dimensional question taxonomy that captures surface-level, implicit, and pragmatic information needs.

The CogQTaxo Framework and Diversification Pipeline

The authors propose a multi-stage reranking algorithm based on CogQTaxo, which classifies questions across three linguistic levels:

  1. Surface Information Need ($T_1$): The explicit question type (e.g., location, person, definition).
  2. Implicit Information Need ($T_2$): Needs required to fill the user's knowledge gap but not explicitly stated.
  3. Pragmatic Expectations ($T_3$): Constraints on the answer, such as specificity, expertise, or objectivity.

The system architecture follows a structured pipeline to ensure both relevance and diversity.

System architecture for question diversification using the CogQTaxo framework.

The pipeline consists of four main steps:

  • Analysis: Extracting content words and predicting CogQTaxo types for the query.
  • Initial Retrieval: Fetching a pool of relevant questions using a standard IR model like Okapi BM25.
  • Reranking: Scoring questions based on a linear combination of IR relevance, type similarity, and type novelty.
  • Greedy Diversification: Iteratively selecting questions that maximize "infoNeeds novelty" to ensure a variety of information types in the final result set.

Experimental Evaluation

The approach was evaluated using a massive crawl of over 1.38 million question-answer postings from Yahoo! Knowledge and Baidu Zhidao. A gold standard evaluation set of 100 queries was manually labeled by human annotators to provide a ground truth for testing.

The researchers compared their full algorithm (Predict_Div) against several baselines, including standard retrieval and a "Bag-of-Words" diversification approach (Bow_Div). Performance was measured using "Intent-Aware" metrics (MAP_IA, MRR_IA, and P@K_IA), which reward systems for covering multiple intents in the top results.

Performance comparison of the proposed Predict_Div model against various baselines using Intent-Aware metrics.

Key Results and Insights

The experimental results highlight several critical findings:

  • Superior Performance: The Predict_Div model consistently outperformed all other methods across all metrics. It achieved a MAP_IA of 0.164, a significant improvement over the standard retrieval baseline of 0.114.
  • Semantic Similarity is Insufficient: High semantic similarity does not guarantee user satisfaction. If a user's intent is ambiguous, providing diverse types of answers is more valuable than variations of the same answer.
  • Failure of Naive Diversification: The "Bag-of-Words" approach (Bow_Div) often performed worse than standard retrieval. This indicates that effective diversification requires understanding the linguistic and pragmatic "type" of the question rather than just word differences.
  • Implicit Needs Matter: By including $T_2$ (Implicit Needs), the system can provide supportive information that the user likely requires but did not explicitly ask for.

Conclusion and Future Work

The CogQTaxo-based diversification approach successfully addresses the redundancy problem in question retrieval by modeling multi-dimensional information needs. While the current evaluation relies on automatically predicted types for the relevance set, future work aims to refine these prediction models and conduct deeper analyses to mitigate potential biases.

Summary updated

BibTeX
@inproceedings{zhang2011diversifying,
  author = {Yaoyun Zhang and Xiaolong Wang and Xuan Wang and Ruifeng Xu and Jun Xu and Shixi Fan},
  title = {{Diversifying Information Needs in Results of Question Retrieval}},
  booktitle = {IJCNLP},
  year = {2011},
}
Figure