Abstract
Content moderation for short video advertising is a dynamic and adversarial task. Existing benchmarks for Vision-Language Models (VLMs) often fail to capture the temporal shifts in content distribution and evolving policy standards. We present AdDriftBench (ADB), the first benchmark specifically designed to evaluate VLM performance under Data Drift (DD) and Label Drift (LD). Our evaluation of 16 open-source VLMs reveals that while model scale correlates with robustness, significant performance gaps remain in handling rapid tactical shifts by advertisers and fine-grained instruction following for updated moderation rules.
Introduction
The moderation of short video advertisements is a high-stakes "cat-and-mouse" game. Unlike static image classification, the distribution of violating content is constantly shifting. This phenomenon manifests in two primary ways: Data Drift (DD), where advertisers modify video styles (e.g., picture-in-picture, deceptive overlays) to evade filters, and Label Drift (LD), where platform community standards are updated, changing the classification of previously acceptable content.
Current Large Vision-Language Models (VLMs) show potential for automating this process, but their ability to generalize across these temporal and rule-based shifts has not been rigorously tested. AdDriftBench addresses this gap by providing a real-world dataset curated from an active short-video platform.
Methodology
To ensure a high-quality and challenging benchmark, the authors employed a three-step curation pipeline:
- Similarity-based Deduplication: Utilizing CLIP embeddings, the team performed inter-video and intra-video deduplication to remove redundant content and optimize token usage for VLM inference.
- Hard Case Screening: A smaller multimodal model was used to identify "hard cases"—videos where the model was uncertain or incorrect. This ensures the benchmark serves as a true stress test for state-of-the-art VLMs.
- Expert Manual Review: A team of professional ad reviewers validated every label against complex platform guidelines to establish a reliable ground truth.
To measure robustness, the study introduces the SDM (Standard Deviation to Mean) ratio. A lower SDM indicates more stable performance over time, signaling better resistance to data drift.
The AdDriftBench Dataset
ADB comprises 3,480 short video ads divided into two main components:
- Data Drift (DD) Portion: 2,280 videos spanning 8 months, covering 6 primary and 12 secondary risk categories (e.g., gambling, gray market).
- Label Drift (LD) Portion: 1,200 videos across 10 risk categories. Each video is tested against "lenient" (old) and "strict" (new) moderation prompts to evaluate instructional plasticity.
Key Insights and Experimental Results
Benchmarking 16 open-source VLMs (including DeepSeek-VL2, InternVL, and Qwen2.5-VL) yielded several critical findings:
- Performance and Scale: Qwen2.5-VL-72B emerged as the top performer, yet it only achieved an average F1-score of 0.43 on DD scenarios. Larger models generally showed higher accuracy and lower SDM scores, indicating that parameter scale is a key factor in drift robustness.
- The ASR Noise Factor: Incorporating Automatic Speech Recognition (ASR) text into prompts unexpectedly degraded performance. The resulting token-level noise likely obscured the specific visual cues necessary for detecting violations.
- RLHF Limitations: Models frequently struggled to identify "illegal" content. This suggests that safety guardrails implemented during Reinforcement Learning from Human Feedback (RLHF) may inadvertently prevent models from correctly labeling prohibited material for moderation purposes.
- Instruction Following in Label Drift: Most models experienced a significant performance drop when transitioning from lenient to strict instructions, highlighting a weakness in fine-grained adherence to evolving rules.
Conclusion
AdDriftBench highlights the significant challenges VLMs face in the adversarial environment of ad moderation. While larger models offer improved stability, the modest F1-scores across the board suggest that current open-source models are not yet fully equipped for the rapid shifts of real-world data and label drift. Future research should focus on improving "instructional plasticity" and developing architectures that can better filter noise from multimodal inputs.
Summary updated