Bridging the Gap between Intrinsic and Perceived Relevance in Snippet Generation

Jing He, Pablo Duboue, Jian-Yun NieCOLING 2012

TL;DR

This paper tackles the problem of generating search result snippets that accurately reflect a document's true relevance to a query. They propose a novel method that explicitly models the consistency between a user's perceived relevance from a snippet and the document's intrinsic relevance, using a gradient boosting decision tree approach. This leads to more informative snippets that improve both user understanding and document retrieval effectiveness.

What Problem Does This Paper Address?

Search engines present users with snippets alongside search results to help them quickly assess a document's relevance. However, existing methods for generating these snippets often fall short. Two main approaches exist: sentence retrieval (simply picking sentences related to the query) and query-biased summarization (summarizing the document for the query). The problem is that these methods can generate snippets that look relevant but point to irrelevant documents (false positives), or conversely, fail to highlight relevant documents (false negatives). This disconnect arises because they don't explicitly consider the user's perceived relevance from the snippet in relation to the document's intrinsic relevance. The authors aim to bridge this gap by ensuring that snippets for more relevant documents are themselves perceived as more relevant.

Methods

The core of the paper's contribution lies in its novel approach to snippet generation. Instead of just focusing on sentence relevance to the query or document, they introduce a constraint: the perceived relevance of a snippet should be consistent with the intrinsic relevance of its document.

Here's a breakdown of their method:

  • Learning-to-Rank Framework: They frame the problem as learning a sentence scoring function within a learning-to-rank paradigm.
  • Pairwise Preferences: They derive training data based on pairwise preferences. If document $d_1$ is intrinsically more relevant to query $q$ than document $d_2$ ($R_i(q, d_1) > R_i(q, d_2)$), then the snippet generated for $d_1$ ($S_1$) should be perceived as more relevant than the snippet for $d_2$ ($S_2$) ($R_p(q, S_1) > R_p(q, S_2)$).
  • Gradient Boosting Decision Trees (GBDT): They employ GBDT, a powerful machine learning technique, to learn the sentence scoring function. GBDT is chosen for its effectiveness in learning-to-rank tasks.
  • Feature Engineering: They utilize four categories of features to score sentences: Query-Sentence Relevance (QSR), Document-Sentence Fidelity Features (DSF), Sentence Informativeness Features (SI), and Query-Document Relevance Features (QDR). Notably, QDR features are less commonly used in query-biased summarization.
  • Iterative Learning: The GBDT model is trained iteratively. In each step, it adjusts the sentence scoring function to better align perceived snippet relevance with intrinsic document relevance, specifically addressing inconsistencies.

Datasets

The authors utilize datasets from the TREC Web track 2010 and 2011.

  • Training Data: 50 queries from TREC 2011.
  • Test Data: 48 queries from TREC 2010.

The document collection used is ClueWeb09B, containing approximately 428 million documents. The relevance judgments are graded (0-3), with higher scores indicating greater relevance. They also mention using a pruned test dataset for evaluation, which is important for fair comparison with baselines where snippets might not be available.

Key Insights and Ideas

  • Bridging the Perceived-Intrinsic Gap is Crucial: The most significant insight is that simply making snippets look relevant isn't enough. The true value lies in ensuring that the snippet's perceived relevance accurately reflects the document's actual relevance. This directly addresses the problem of misleading users.
  • Consistency as a Learning Objective: By formulating snippet generation as a problem of learning a sentence scoring function that enforces consistency between perceived and intrinsic relevance, they create a more robust and informative system.
  • Query-Document Relevance Features Matter: The inclusion and demonstrated importance of Query-Document Relevance (QDR) features suggest that understanding the overall relationship between the query and the document is as vital as the sentence's direct relevance to the query.
  • Snippets Enhance Document Ranking: Beyond just improving snippet quality, the paper shows that these informative snippets can significantly boost document retrieval performance when incorporated into ranking functions.

Key Results

The paper presents strong experimental results demonstrating the effectiveness of their proposed method:

  • Snippet Quality Improvement: Their method achieved higher scores than both a commercial search engine (SE) and a query-biased summarizer (SUM) across most measures, including Precision (P), F1 Score, Accuracy (ACC), Negative Recall (NR), and Geometric Mean (GM).
  • F1 Score: Improved by 14.0% compared to SE and 7.7% compared to SUM.
  • Precision: Improved by 28.9% compared to SE and 4.9% compared to SUM.
  • Negative Recall: Significantly improved, indicating better identification of irrelevant documents.
  • Document Retrieval Enhancement: When snippets were integrated into a language modeling approach for document retrieval, their method (along with SUM) significantly improved retrieval effectiveness.
  • MAP: Improved by 2.4% compared to the baseline (NO snippets).
  • ERR@10: Improved by 8.3% compared to the baseline.
  • NDCG@10: Improved by 6.5% compared to the baseline.
  • Qualitative Examples: The paper includes examples showing how their snippets more accurately indicate document relevance, avoiding false positives and false negatives seen in SE and SUM snippets.

Limitations and Future Work

  • Approximation of Perceived Relevance: The authors acknowledge that in their implementation, perceived relevance was approximated by query-sentence similarity. They suggest that using actual human-assessed perceived relevance could be a future direction.
  • Candidate Sentence Selection: While their method uses a simple approach to select candidate sentences for training, they propose that incorporating sentence fidelity to the document and direct query relevance into this selection process could further improve results.
  • Sophisticated Retrieval Models: The current document retrieval experiment used a bag-of-word model, which might not fully capture the subtle differences introduced by their improved snippets. Future work could explore more sophisticated retrieval models (e.g., phrase-based) to better leverage these differences.
  • Other Learning-to-Rank Methods: While GBDT proved effective, the authors mention that other pairwise learning-to-rank methods, such as SVM-rank, could also be explored.

Summary updated

BibTeX
@inproceedings{he2012bridging,
  author = {Jing He and Pablo Duboue and Jian-Yun Nie},
  title = {{Bridging the Gap between Intrinsic and Perceived Relevance in Snippet Generation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure