Clickthrough Log Analysis by Collaborative Ranking

Bin Cao, Dou Shen, Kuansan Wang, Qiang Yang

Introduction

Search engines rely heavily on clickthrough logs to understand user intent and relevance. However, these logs are inherently noisy due to position bias—where users click top results regardless of relevance—and data sparseness, particularly in the "long tail" of rare queries. Traditional models often focus on "co-clicks" but ignore the valuable information in "skips"—results that were presented but not clicked. This paper proposes Collaborative Ranking (CoRank), a framework that addresses these issues by combining pairwise preference learning with matrix factorization.

Methodology

The CoRank framework unifies "learning to rank" with latent factor models to create a robust representation of query-URL relevance.

Pairwise Preferences

To mitigate position bias, the model shifts from absolute click counts to relative judgments. If a user clicks a lower-ranked URL $j$ while skipping a higher-ranked URL $k$, it implies a preference $j \succ k$. This approach captures the user's intent more accurately than treating clicks in isolation.

Pairwise preference extraction from clickthrough logs

Matrix Factorization and Optimization

To handle data sparseness, CoRank maps queries and URLs into a shared latent factor space. This allows the model to infer relevance for rare queries by leveraging patterns from similar queries. The optimization uses the Bradley-Terry model to define preference likelihood, maintaining linear computational complexity relative to the number of preference pairs, ensuring scalability for production-level datasets.

Hybrid Model

The authors noted that CoRank and traditional random walk models (which focus on clicks) provide orthogonal information. By combining them into a linear hybrid model, the strengths of both "click" and "skip" signals are harnessed.

Experimental Evaluation

The model was tested on a massive dataset from a commercial search engine, comprising over 1.6 million queries and nearly 10 million URLs. Performance was measured by the accuracy of predicted pairwise preferences on a held-out test set.

Results

The experimental results demonstrate the superiority of the CoRank and Hybrid approaches over standard random walk models.

Comparison of prediction accuracy across different models

The Hybrid model achieved an accuracy of 0.94, a significant improvement over individual models. This confirms that collaborative ranking and click-based reinforcement are highly complementary.

Conclusion and Future Work

CoRank effectively mines the "long tail" of search logs by focusing on relative user preferences and latent factors. While it significantly improves ranking accuracy, future work will address the "cold start" problem for new queries and URLs, and explore the integration of content-based features and confidence weighting for different types of clicks.

Summary updated

Abstract

Analyzing clickthrough log data is important for improving search performance as well as understanding user behaviors. In this paper, we propose a novel collaborative ranking model to tackle two difficulties in analyzing clickthrough log. First, previous studies have shown that users tend to click top-ranked results even they are less relevant. Therefore, we use pairwise ranking relation to avoid the position bias in clicks. Second, since click data are extremely sparse with respect to each query or user, we construct a collaboration model to eliminate the sparseness problem. We also find that the proposed model and previous popular used click-based models address different aspects of clickthrough log data. We further propose a hybrid model that can achieve significant improvement compared to the baselines on a large-scale real world dataset.
BibTeX
@inproceedings{cao2010clickthrough,
  author = {Bin Cao and Dou Shen and Kuansan Wang and Qiang Yang},
  title = {{Clickthrough Log Analysis by Collaborative Ranking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2010},
  doi = {10.1609/aaai.v24i1.7579},
}
Figure