Simulating User Watch-Time to Investigate Bias in YouTube Shorts Recommendations

S. Dağtaş, Mert Can Çakmak, Nitin AgarwalarXiv 2025 · arXiv

TL;DR

Researchers simulated user behavior on YouTube Shorts to reveal how the algorithm reacts to "watch-time" signals. They found that even small increases in viewing duration trigger a powerful feedback loop that rapidly narrows content diversity, though the system struggles to maintain topical relevance for specific, complex geopolitical conflicts compared to broader themes.

What Problem Does This Paper Address?

While we understand a fair amount about how long-form YouTube recommendations work, short-form video (SFV) platforms like YouTube Shorts—which boasts over 1.5 billion monthly users—operate on a different logic. SFV relies on "passive engagement": rapid swiping and varying watch durations rather than active clicks or searches.

The authors address a critical gap: how do these lightweight engagement cues (like watching a 60-second clip vs. skipping after 3 seconds) influence the "recommendation chain"? Specifically, they investigate whether these algorithms exhibit bias by drifting away from specific topics or amplifying certain narratives, particularly in the sensitive arena of global geopolitics.

Methods

To study a "black box" algorithm without access to internal data, the researchers built a custom simulation framework:

  1. Automated Scraper: Since the official YouTube API doesn't support Shorts recommendations, they developed a Selenium-based scraper using headless Chrome. To ensure a "clean slate," they used incognito sessions and cleared cookies for every run.
  2. Simulated Behavior: They tested two conditions across 50-step recommendation chains:
  3. Baseline: A "bored" user who skips every video after 3 seconds.
  4. Interest-based: A user whose watch time (3, 15, or 60 seconds) is dynamically determined by how relevant the video title is to a specific seed topic.
  5. LLM-as-a-Judge: To scale the analysis of 400,000+ videos, they used GPT-4o to score the relevance of video titles on a scale of 0 to 2. They validated this by benchmarking against LLaMA 3 and Gemini 1.5, finding GPT-4o to be the most accurate (78.73% accuracy on validation sets).

Datasets

The study focused on four thematic datasets chosen for their geopolitical sensitivity and complexity:

  • Broader Russia: General influence, BRICS, and foreign policy.
  • Russia–Ukraine War: Specific military and humanitarian developments.
  • Broader China: Global expansion and the "Belt and Road" initiative.
  • South China Sea Dispute: Specific maritime and territorial conflicts.

The researchers collected 1,000 seed videos per topic, resulting in a massive dataset of 404,000 Shorts across the recommendation chains.

Key Insights and Ideas

  • The Amplification Effect: The algorithm treats even minor differences in watch time as high-confidence signals. Watching a video for 60 seconds instead of 3 doesn't just "nudge" the algorithm; it causes a persistent shift in the entire recommendation path.
  • Topic Generalization (The "Drift"): The algorithm is much better at keeping users within broad categories (e.g., "Russia") than specific ones (e.g., "Russia-Ukraine War"). When users showed interest in specific conflicts, the algorithm often drifted toward broader, more "viral" or generalized content within that geographic region.
  • Low "Ceiling" for Relevance: Surprisingly, even in the "interest-based" sessions, the average relevance score rarely exceeded 0.35 on a 2.0 scale. This suggests that the Shorts feed is inherently noisy, delivering a high volume of "filler" content that is only marginally related to the user's apparent interests.
  • Feedback Loops vs. Diversity: The system prioritizes reinforcement over exploration. Once a user shows interest, the "narrowing" of content happens almost immediately, potentially creating echo chambers much faster than in long-form video ecosystems.

Key Results

The researchers used Area Under the Curve (AUC) to measure cumulative relevance across the 50-video recommendation depth.

  • Significant Gains in Broad Topics: For "Broader Russia," the AUC jumped from 8.10 (baseline) to 15.92 (interest-based), a statistically massive shift ($p = 2.42 \times 10^{-46}$).
  • Struggles with Specificity: In contrast, the "South China Sea" topic saw a much smaller gain (from 5.06 to 5.75). This indicates the algorithm is less responsive to interest when the topic is niche or highly specific.
  • Early Emergence: The divergence between the "skipping" user and the "interested" user appeared very early in the chain (often within the first 5-10 videos) and persisted through the 50th recommendation.

Limitations and Future Work

The authors acknowledge several limitations:

  • Metadata Constraints: The study relied primarily on video titles. Because Shorts often have "clickbaity" or minimal titles, the LLM might miss relevance that is only apparent in the audio or visual content.
  • External Confounders: Factors like the time of day, regional popularity, and "trending" status were not explicitly controlled and likely influenced the results.
  • Single Signal: The simulation only looked at watch-time. Real users also "like," "subscribe," and "comment," which are likely even stronger algorithmic signals.

Future research aims to incorporate these additional engagement signals and expand the topical scope beyond geopolitics to see if these "drift" and "generalization" patterns hold true for lifestyle or entertainment content.

Summary updated

Abstract

Short-form video platforms such as YouTube Shorts increasingly shape how information is consumed, yet the effects of engagement-driven algorithms on content exposure remain poorly understood. This study investigates how different viewing behaviors, including fast scrolling or skipping, influence the relevance and topical continuity of recommended videos. Using a dataset of over 404,000 videos, we simulate viewer interactions across both broader geopolitical themes and more narrowly focused conflicts, including topics related to Russia, China, the Russia-Ukraine War, and the South China Sea dispute. We assess how relevance shifts across recommendation chains under varying watch-time conditions, using GPT-4o to evaluate semantic alignment between videos. Our analysis reveals patterns of amplification, drift, and topic generalization, with significant implications for content diversity and platform accountability. By bridging perspectives from computer science, media studies, and political communication, this work contributes a multidisciplinary understanding of how engagement cues influence algorithmic pathways in short-form content ecosystems.
BibTeX
@misc{dagtas2025simulating,
  author = {S. Dağtaş and Mert Can Çakmak and Nitin Agarwal},
  title = {{Simulating User Watch-Time to Investigate Bias in YouTube Shorts Recommendations}},
  year = {2025},
  eprint = {2507.04534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.04534},
  howpublished = {arXiv},
}
Figure