Are Web User Comments Useful for Search?

Wai Gen Yee, Andrew Yates, Shizhu Liu, Ophir Frieder

Abstract

This study investigates the utility of user-generated comments in enhancing video search performance on platforms like YouTube. While often dismissed as noisy or redundant, comments provide a rich source of descriptive metadata frequently omitted by uploaders. By employing Kullback-Leibler (KL) divergence for content measurement and document-centric pruning for noise reduction, the researchers demonstrate that indexing comments can improve search accuracy (Mean Reciprocal Rank) by up to 15%, particularly for popular videos where comment volume is high.

Introduction

Traditional video search engines rely heavily on uploader-provided metadata, such as titles, descriptions, and tags. However, this metadata is often sparse or idiosyncratic. For instance, a video of a "dog show" might lack the keyword "dog" in its official fields, rendering it unsearchable for that specific term. User comments represent a massive, albeit noisy, repository of additional descriptive text. This paper evaluates whether the informational value of these comments outweighs the technical costs of indexing them.

Methodology

The researchers framed search as a "known-item" retrieval task. The methodology involved:

  1. Information Content Measurement: KL divergence was used to quantify how effectively different text fields distinguished videos.
  2. Query Simulation: Search queries were simulated using uploader keywords (tags) that were removed from the index to test if the system could retrieve the video using only titles, descriptions, and comments.
  3. Evaluation: Performance was measured using Mean Reciprocal Rank (MRR).
  4. Index Optimization: To mitigate noise (e.g., "LOL", "First!"), the authors applied document-centric pruning, which ranks and retains only the terms contributing most to a document's information content.

Data and Temporal Dynamics

The study utilized four datasets crawled from YouTube in 2009: rand3500 and rand10K (representing the "long tail") and pop500 and pop1500 (representing popular content). A key finding was the rate at which these videos become "searchable" via comments.

Figure 2: Accumulation of comments over time for popular vs. random videos.

As shown in Figure 2, popular videos accumulate comments rapidly, reaching a threshold of utility for search within approximately one day. In contrast, random videos may take two weeks or longer to accrue sufficient descriptive data.

Results and Optimization

The inclusion of comments significantly boosted search accuracy. For popular videos, MRR improved by 9% to 20%, with an overall dataset improvement of approximately 15%.

Crucially, the researchers found that indexing the entire comment thread is unnecessary and often counterproductive due to noise.

Figure 3: Impact of comment field pruning on Mean Reciprocal Rank (MRR) percentage change.

Figure 3 illustrates that pruning up to 50% of the comment text—keeping only the most informative terms—not only reduces index size by 70% but actually improves search accuracy by filtering out low-value "fluff."

Conclusion and Limitations

While user comments are a powerful tool for improving search, challenges remain. "Eager matching" can cause popular videos with many comments to overshadow newer, highly relevant content. Additionally, the study does not fully address the impact of malicious comment spam or exploratory search behaviors. Future research should focus on temporal weighting and robust spam detection to further refine comment-based indexing.

Summary updated

BibTeX
@inproceedings{yee2009are,
  author = {Wai Gen Yee and Andrew Yates and Shizhu Liu and Ophir Frieder},
  title = {{Are Web User Comments Useful for Search?}},
  booktitle = {LSDS-IR @ SIGIR},
  year = {2009},
}
Figure