ClipMind: A Framework for Auditing Short-Format Video Recommendations Using Multimodal AI Models

Aoyu Gong, Sepehr Mousavi, Yiting Xia, Savvas ZannettouICWSM 2025 · DOI

TL;DR

This paper introduces ClipMind, a novel framework for auditing AI-driven recommendation systems on short-format video platforms like TikTok. By leveraging state-of-the-art multimodal AI models, ClipMind can analyze video content and user interactions at scale, offering insights into how these algorithms shape user feeds and potentially lead to "rabbit holes" of content.

What Problem Does This Paper Address?

Social media platforms are increasingly relying on AI recommendation systems to curate personalized feeds, especially for short-form videos. While this enhances user engagement, it also raises concerns about potential negative consequences like online polarization and user radicalization. Existing methods for auditing these systems are often manual, time-consuming, and lack scalability. This paper addresses the need for an automated, generalizable, and scalable framework to understand and audit these complex recommendation algorithms, particularly on platforms like TikTok and YouTube Shorts.

Methods

ClipMind employs a multi-stage approach using state-of-the-art (SOTA) AI models:

  1. Generating Embeddings: It extracts multimodal features from videos, including visual and audio content (using ImageBind), LLM-generated descriptions (using Video-LLaMA and ADA-002), user-defined metadata (titles and hashtags, processed by ADA-002), and LLM-generated keywords (using GPT-4 and ADA-002). These features are combined into a single, comprehensive embedding for each video.
  2. Assessing Video Similarity: Cosine similarity is used to measure the semantic similarity between video embeddings. The framework identifies the optimal combination of features and a similarity threshold for accurate similarity assessment through a rigorous evaluation process involving manual annotation.
  3. Video Sequence Analysis: A sliding window approach is applied to analyze sequences of recommended videos. Within each window, videos are represented as a graph where nodes are videos and edges connect similar videos. Connected components in these graphs are identified, and their associated keywords are analyzed to understand the thematic coherence of recommended content.

Datasets

The study utilizes a dataset collected from TikTok. This dataset consists of five traces: one random trace and four topic-specific traces (Food, Beauty Care, Mental Health, and War/Military). Each trace contains a sequence of 500 recommended videos. The data collection involved automated accounts (bots) that simulated user behavior to generate these recommendation sequences.

Key Insights and Ideas

  • Scalable Auditing: ClipMind eliminates the need for manual annotation, making it a scalable solution for auditing recommendation systems.
  • Multimodal Understanding: By integrating visual, audio, text, and metadata, ClipMind captures a richer semantic understanding of short-form videos than text-only approaches.
  • No Evidence of Extreme Rabbit Holes (for Niche Topics): For niche topics like Mental Health and War/Military, the analysis did not find evidence of the algorithm pushing users into "informational rabbit holes" of extreme content.
  • Mainstream Topics Show Less Diversity: When users show interest in mainstream topics like Food and Beauty Care, the recommendation algorithm tends to recommend a high percentage of similar videos, potentially leading to less content diversity.
  • Framework Generalizability: The modular design and reliance on SOTA AI models suggest ClipMind can be adapted to other short-form video platforms.

Key Results

  • Mainstream Topic Promotion: For Food and Beauty Care topics, a significant percentage of videos within a sliding window (60-80%) were related to the topic of interest.
  • Niche Topic Suppression: For Mental Health and War/Military topics, a much smaller percentage of videos within a window were related to the niche topic, comparable to the random trace.
  • Optimal Feature Combination: The best performing feature combination for assessing video similarity involved visual content, user-defined metadata, and LLM-generated keywords.
  • High Performance: The framework achieved high performance metrics (e.g., 0.9580 accuracy, 0.8923 F1 score) in identifying similar videos.
  • Balance Measurement: A metric called "balance measurement" (β) was introduced to quantify the promotion of mainstream versus suppression of niche topics, showing that Food and Beauty Care were promoted more than Mental Health and War/Military.

Limitations and Future Work

  • Platform Specificity: The current demonstration is limited to TikTok. Future work aims to expand the analysis to other platforms like YouTube Shorts and Instagram Reels.
  • Limited Topic Scope: The analysis covered a small number of topics. Future research will investigate a broader range of topics, including political content and entertainment.
  • Snapshot Analysis: The current study represents a single snapshot of the TikTok algorithm. Longitudinal audits are needed to account for the dynamic nature of these algorithms.
  • Bias and Diversity: The study did not explicitly measure biases or evaluate content diversity. Future work plans to customize keyword pools to analyze specific domains and culinary traditions.
  • Reliance on Closed-Source Models: ClipMind currently uses closed-source AI models, which incur costs. The authors plan to develop an "OpenClipMind" variant using only open-source models.

Summary updated

BibTeX
@inproceedings{gong2025clipmind,
  author = {Aoyu Gong and Sepehr Mousavi and Yiting Xia and Savvas Zannettou},
  title = {{ClipMind: A Framework for Auditing Short-Format Video Recommendations Using Multimodal AI Models}},
  booktitle = {ICWSM},
  year = {2025},
  doi = {10.1609/icwsm.v19i1.35838},
}
Figure