TL;DR
Current text-video retrieval models exhibit a systematic preference for AI-generated videos over real ones, a phenomenon the authors term Visual-Temporal Induced Source Bias. By building a new evaluation benchmark and analyzing model embeddings, this paper uncovers how generative models leave a distinct mathematical "fingerprint" that tricks retrieval algorithms, and proposes a contrastive learning framework to successfully mitigate this bias.
What Problem Does This Paper Address?
As AI-generated content (AIGC) floods the internet, our digital ecosystem is shifting. While researchers have previously noted that text and image search engines tend to favor AI-generated content (known as "source bias"), the video domain has remained largely unexamined.
Video retrieval is inherently more complex than text or image retrieval because it requires models to process both static visual features and dynamic temporal changes over time. This paper investigates whether popular text-video retrieval models unfairly prioritize AI-generated videos over real-world videos of identical semantic meaning, what causes this preference, and how we can fix it.
Methods
To isolate and study this bias without semantic confounding, the authors developed a multi-step framework:
- Semantic-Equivalent Video Generation: To ensure that retrieval models weren't simply choosing AI videos because they matched the text query better, the authors used GPT-4 to fuse multiple descriptive captions of real videos into a single comprehensive prompt. They then fed these prompts into state-of-the-art video generators to create AI counterparts with identical underlying semantics.
- Robust Bias Metrics: The authors introduced two novel metrics:
NormalizedΔ: A metric that isolates pure source bias by subtracting positional baseline differences (LocationΔ) from actual retrieval differences (RelativeΔ), removing the impact of minor semantic discrepancies.MixR: A holistic metric combining Recall@1 (R@1), Mean Rank (MeanR), and Median Rank (MedR) to evaluate the entire retrieval list rather than just the top result.- Ablation via Shuffling and Single-Frames: The authors isolated the root causes of the bias by disrupting temporal data (shuffling video frames randomly or in reverse) and isolating visual data (performing retrieval using only a single static frame).
- Contrastive Debiasing: The authors fine-tuned retrieval models using a triplet contrastive loss function. This explicitly penalized the model when it assigned a higher similarity score to an AI-generated video than to its corresponding real video for a given text query.
Datasets
The authors constructed a new benchmark based on the widely used MSR-VTT dataset (which contains 10,000 real-world videos with 20 English captions each).
Using two open-source video generation models—CogVideoX and OpenSora V1.2—they generated a total of 13,000 AI videos. The benchmark includes:
- A Training Set: 9,000 text-conditioned videos generated via OpenSora.
- Four Test Sets (1,000 videos each): Designed around different generation constraints:
- TextCond (CogVideoX & OpenSora): Generated purely from text prompts.
- ImageCond (OpenSora): Generated using text prompts combined with a real video keyframe.
- VideoExt (OpenSora): Generated by extending the first half of a real video.
Key Insights and Ideas
- The Illusion of Temporal Simplicity: By measuring optical flow entropy, the authors discovered that real videos possess much richer, more chaotic temporal depth. AI-generated videos, by contrast, feature simplified frame-to-frame transitions. Ironically, video encoders mistake this lack of temporal depth for a "cleaner," higher-relevance signal, ranking AI videos higher.
- Training Poisoning: If AI-generated videos are mixed into a retriever's training dataset, the bias compounds drastically. Even a modest 20% inclusion of AI videos in the training set causes the model to aggressively favor AI content during testing.
- The "Generative Ghost" Fingerprint: When visualizing the mathematical shift between the original biased models and the debiased models using t-SNE, the authors found that the transformation vectors ($p$-vectors) formed a highly concentrated cluster. This proves that AI video generators leave a universal, highly consistent mathematical artifact—a "ghost"—across entirely different video contents.
Key Results
- Widespread Bias Confirmed: Across three prominent off-the-shelf retrieval models (ALPRO, Frozen in Time, and InternVideo), both
RelativeΔandNormalizedΔwere consistently negative across all text-conditioned datasets, proving a systemic preference for AI videos. - Successful Debiasing: The proposed contrastive learning approach successfully reversed the bias. For example, on the OpenSora TextCond dataset, the
NormalizedΔfor InternVideo's R@1 shifted from a negative bias to a positive score of 200.73, meaning the model successfully prioritized real videos at the top of the rank list. - Universal Application: Adding the average extracted debiasing vector ($p_{avg}$) directly to raw, real video representations successfully reduced retrieval bias across all metrics, proving the vector captures a generalizable property of AIGC video encoders.
Limitations and Future Work
The authors acknowledge a few limitations in their current work:
- Open-Source vs. Commercial Models: The benchmark primarily relies on open-source generators (OpenSora, CogVideoX). While a small 100-video validation test using the commercial platform PIKA yielded consistent results, broader testing on closed-source giants like OpenAI's Sora or Runway Gen-3 is needed.
- Resolution and Frame Rates: Extreme downsampling (e.g., to 50x50 pixels) degrades video quality enough to attenuate the bias, indicating that visual resolution plays a major role in how encoders perceive these hidden features.
- Future Directions: The team plans to investigate more sophisticated retrieval architectures and explore whether generative models overemphasize specific keywords in text prompts, further driving the ranking bias.
Summary updated