Reweighting Clicks with Dwell Time in Recommendation

Ruobing Xie, Lin Ma, Shaoliang Zhang, Feng Xia, Leyu Lin

TL;DR

This paper introduces a framework to move beyond binary "click" signals in recommendation systems by reweighting them using dwell time. By defining a new "valid read" behavior and applying a novel sigmoid-based normalization function, the authors successfully filtered out clickbait and improved user satisfaction in a massive real-world deployment at WeChat.

What Problem Does This Paper Address?

In modern recommender systems, the "click" is the gold standard for positive feedback. However, clicks are inherently noisy. A user might click on a "clickbait" headline only to leave immediately when the content doesn't match the title. Standard models treat this dissatisfied bounce the same as a deep, meaningful read because both are binary "1s" in a training set.

While dwell time (the duration a user spends on an item) is a known proxy for satisfaction, it presents two major challenges:

  1. Bias toward long content: Directly optimizing for raw dwell time unfairly favors long-form videos or articles, potentially starving "light" users or short-form content creators.
  2. Diminishing returns: The difference in satisfaction between spending 1 second vs. 15 seconds is massive, whereas the difference between 601 seconds vs. 615 seconds is negligible.

Methods

The authors propose a Click Reweighting framework consisting of two main components:

1. Valid Read Selection

Instead of treating every click as a positive label, they define a "valid read" based on three criteria to ensure fairness across different users and items:

  • T1 (Global Threshold): Dwell time exceeds a common-sense threshold (approx. 15s).
  • T2 (Light User Patch): Clicks from users who rarely interact (less than 7 clicks/week) are preserved to avoid losing their sparse data.
  • T3 (Relative Item Threshold): Clicks that fall within the top 90% of dwell times for that specific item, which protects high-quality short-form content.

2. Normalized Dwell Time (NDT) Function

To quantify the "intensity" of a click, they map raw dwell time ($T$) to a weight ($T_N$) using a specialized sigmoid function: $$T_N = \frac{A}{1 + \exp(-\frac{T - offset}{\tau})} - B$$

The sigmoid NDT function (right) is designed to be highly sensitive near the valid read threshold and plateau for very long durations.

This function is designed with two key properties:

  • Sensitivity (C1): The gradient is steepest near the "valid read" threshold ($offset$), helping the model clearly distinguish between a bounce and a real read.
  • Saturation (C2): The curve flattens for very long dwell times, preventing "heavy" users or exceptionally long items from dominating the training process.

3. Multi-Task Learning (MTL)

The final model uses a shared-bottom architecture with two MLP towers: one predicts the probability of a "valid read," and the other predicts the weighted valid read probability.

Datasets

The model was evaluated on a massive dataset from WeChat Top Stories:

  • Scale: 29.7 million users, 5.3 million items, and 751 million total instances.
  • Composition: The dataset included 104 million clicks, of which 89.6 million were classified as "valid reads."
  • Evaluation: Both offline AUC metrics and a 7-day online A/B test involving 5 million users.

Key Insights and Ideas

  • Information Gain vs. Time: The core philosophy is that a recommender should help users get information efficiently. More time spent isn't always better; more valid time is.
  • The Log-Normal Distribution: The authors observed that log dwell time roughly follows a Gaussian distribution. This insight allowed them to set their $offset$ and saturation points ($x_h$) based on standard deviations ($\mu \pm \sigma$) of the global distribution.
  • Efficiency over Duration: Interestingly, the online A/B test showed that while total dwell time (DT) slightly decreased, the number of clicks (ACN) and impressions (AIN) increased. This suggests users were finding high-quality content faster and consuming more of it, rather than getting "stuck" on a few long items.

Key Results

The proposed VR+NDT (Valid Read + Normalized Dwell Time) model outperformed all baselines:

  • Offline: Achieved a 5.62% Relative Improvement in valid read prediction over a standard CTR model.
  • Online A/B Test:
    • CTR: +2.577%
    • Average Click Number (ACN): +4.139%
    • Average Impression Number (AIN): +1.545%
Table 1: Offline evaluation results showing the superiority of the VR+NDT approach.

The "Dwell Time Migration" analysis (Figure 3) confirmed that the model successfully increased dwell time for short-duration behaviors (P10/P20 quantiles) while reducing the over-emphasis on extremely long-duration behaviors.

Figure 3: Dwell time migration on different dwell time quan- tiles and different user activeness in an online system.

Limitations and Future Work

While effective, the authors note that the current "valid read" definition is relatively straightforward. Future work could involve:

  • Sophisticated Modeling: Exploring more complex ways to define valid reads beyond fixed thresholds and quantiles.
  • Long-term Metrics: Investigating the impact of click reweighting on long-term user retention and ecosystem health.
  • Feature Engineering: Better encoding of content features specifically for dwell time prediction, rather than relying on shared features with CTR towers.

Summary updated

Abstract

The click behavior is the most widely-used user positive feedback in recommendation. However, simply considering each click equally in training may suffer from clickbaits and title-content mismatching, and thus fail to precisely capture users' real satisfaction on items. Dwell time could be viewed as a high-quality quantitative indicator of user preferences on each click, while existing recommendation models do not fully explore the modeling of dwell time. In this work, we focus on reweighting clicks with dwell time in recommendation. Precisely, we first define a new behavior named valid read, which helps to select high-quality click instances for different users and items via dwell time. Next, we propose a normalized dwell time function to reweight click signals in training for recommendation. The Click reweighting model achieves significant improvements on both offline and online evaluations in real-world systems.
BibTeX
@inproceedings{xie2022reweighting,
  author = {Ruobing Xie and Lin Ma and Shaoliang Zhang and Feng Xia and Leyu Lin},
  title = {{Reweighting Clicks with Dwell Time in Recommendation}},
  booktitle = {The Web Conference},
  year = {2022},
}
Figure