Answers, not links: extracting tips from yahoo! answers to address how-to web queries

Ingmar Weber, Antti Ukkonen, Aristides GionisWSDM 2012 · DOI

TL;DR

This paper tackles the challenge of providing direct, actionable "tips" as answers to "how-to" web queries, moving beyond traditional link-based search results. By mining Yahoo! Answers, they develop methods to identify how-to queries, extract high-quality tips, and match them effectively, aiming to enhance the user experience for practical information seeking.

What Problem Does This Paper Address?

Search engines are the primary gateway to online information, and while they've evolved to offer direct answers for many query types (weather, stocks, etc.), a significant gap remains for "how-to" queries. Users often seek concrete, step-by-step advice rather than just links to potentially relevant web pages. This paper addresses the problem of automatically identifying these "how-to" queries, extracting concise, actionable "tips" from a large question-answering platform like Yahoo! Answers, and then serving the most relevant tip directly to the user. The core challenge lies in filtering out low-quality or irrelevant information from a vast, user-generated source and accurately matching it to user intent.

Methods

The authors propose a modular system to address this problem, breaking it down into three key stages:

  1. How-to Query Detection:
    • Literal Approach: Initially, they considered simple keyword matching for queries starting with "how to," "how do I," or "how can I." While precise, this has low recall.
  2. Verb-First Approach: They expanded this by looking for queries starting with an English verb in its infinitive form, recognizing that experienced users might phrase queries more directly.
  3. Generalization: To improve recall, they generalized literal "how to" queries by removing the prefix and considering the remaining phrase as a potential how-to intent. To mitigate false positives (e.g., "google" becoming a how-to query), they used a threshold based on the fraction of times the normalized query appears with a "how to" prefix.
  4. Site-Based Approach: They also leveraged click data from known "how-to" websites (like ehow.com) to identify queries that, while not explicitly "how-to," frequently lead to such sites.
  5. Tip Extraction and Filtering:
    • Candidate Tip Generation: Tips are extracted from Yahoo! Answers, specifically from (question, best answer) pairs. They filter for English content, literal "how to" questions, and answers starting with a verb. The question title is transformed into the "goal" (X) and the answer into the "suggestion" (Y) for a tip formatted as "To X: Y."
    • Quality Assessment: A crucial step is filtering out low-quality tips. They used crowdsourcing (Mechanical Turk) to label 20,000 candidate tips as "very good," "ok," or "bad."
    • Machine Learning Classifier: Based on these labels, they trained machine learning models (SVM, decision tree, k-NN) using various features (term frequencies, latent semantic topics, textual properties like reading level and sentiment) to classify tips as "very good" versus "not very good." The focus was on high precision to avoid showing bad tips.
  6. Tip Matching and Ranking:
    • Query-Tip Matching: They developed algorithms to match identified how-to queries to extracted tips. This involved defining a "goal" in the tip and matching it to the query. They experimented with AND/OR modes and a "span" parameter (fraction of query terms appearing in the tip goal) to balance precision and recall.
  7. Ranking: For queries with multiple matching tips, a simple ranking function was used, prioritizing tips where the query terms appeared in the tip's goal, normalized by the goal's length. Ties were broken using tf-idf and tip length.

Datasets

  • Query Log: A large sample of query logs from a commercial search engine (May 2010 - January 2011) was used for identifying how-to queries.
  • Yahoo! Answers Dataset: All (question, best answer) pairs from Yahoo! Answers posted between May 2006 and August 2010 were used for tip extraction.
  • Crowdsourced Labels: A benchmark of 20,000 candidate tips were manually labeled by crowdworkers for quality assessment, used for training and evaluating the tip quality classifier.

Key Insights and Ideas

  • "How-to" is a Significant Query Intent: The paper quantifies that a non-trivial percentage (around 2%) of search query volume has a "how-to" intent, underscoring the potential value of direct tip delivery.
  • Beyond Literal "How-To": Simply looking for "how to" is insufficient. Users express their need for instructions in more varied ways, necessitating more sophisticated query detection methods.
  • Quality is Paramount for Direct Answers: When providing a direct answer, especially on limited screen real estate (like mobile), showing a low-quality or irrelevant tip is worse than showing no tip at all. This drove the focus on high-precision filtering.
  • Tip Structure Matters: The "To X: Y" structure from Yahoo! Answers proved beneficial for both extraction and matching, making the process more manageable and the tips more user-friendly.
  • Modularity is Key: The system is designed modularly, allowing individual components (query detection, tip extraction, filtering, matching) to be improved independently.

Key Results

  • How-to Query Volume: Approximately 2.0% of total search volume was identified as having a clear "how-to" intent.
  • Tip Quality Classification: The machine learning classifier achieved a precision of 0.94 for separating "very good" tips from "bad" ones (over 10-fold cross-validation).
  • Query-Tip Matching: The matching algorithm could provide a tip for 4.5% of "how-to" queries with high precision (0.92), and coverage could be doubled to 10.0% at the cost of precision (0.75).
  • Crowdsourced Tip Quality: Of the 249,675 candidate tips generated, 25% were judged "very good," 48% "ok," and 27% "bad" by human evaluators.

Limitations and Future Work

  • Tip Extraction Bias: The current tip extraction relies on Yahoo! Answers' specific structure and content. Future work could explore extracting tips from other sources like Twitter or blogs, potentially using hashtags or verb-based rules.
  • Query Intent Generalization: While they improved beyond literal "how to," the paper acknowledges that more advanced techniques could further enhance how-to query identification.
  • Matching Granularity: The current matching focuses on the "goal" of the tip. Future work could explore more nuanced matching strategies.
  • User Experience: The paper suggests evaluating user studies to understand how users react to serendipitous tips that might not directly match a "how-to" intent but offer interesting advice (e.g., a "Did you know...?" flavor).
  • Tip Freshness: The current system doesn't consider the age or recency of a tip, which could be important for certain domains (e.g., software tips).

Summary updated

BibTeX
@inproceedings{weber2012answers,
  author = {Ingmar Weber and Antti Ukkonen and Aristides Gionis},
  title = {{Answers, not links: extracting tips from yahoo! answers to address how-to web queries}},
  booktitle = {WSDM},
  year = {2012},
  doi = {10.1145/2124295.2124369},
}
Figure