TL;DR
The authors introduce GeneVA, a large-scale dataset containing over 16,000 AI-generated videos coupled with rich, human-annotated spatio-temporal artifact labels. Unlike previous benchmarks, GeneVA provides both fine-grained bounding boxes and free-form textual descriptions of video glitches, enabling the training of a baseline automated artifact detector that generalizes to unseen generative models.
What Problem Does This Paper Address?
While text-to-video (T2V) diffusion models have made staggering leaps in generating dynamic content, they frequently suffer from jarring visual bugs. These range from spatial defects (like distorted anatomy) to temporal anomalies (like objects randomly popping out of existence or violating basic laws of physics).
Standard automated metrics like Fréchet Video Distance (FVD) often correlate poorly with actual human perception of these glitches. To fix this, researchers need human-annotated data. However, existing video quality datasets only provide holistic, video-level scores. They don't specify where or when an artifact occurs, nor do they describe its nature, leaving a major gap in our ability to systematically diagnose and mitigate T2V errors.
Methods
The researchers built a comprehensive pipeline to collect, annotate, and model video artifacts:
- Prompt Selection & Generation: They sampled 5,452 diverse, real-world prompts from the VidProM dataset using kernel herding on text embeddings to avoid semantic bias. They then generated videos using three prominent T2V models: OpenAI's Sora, Pika, and VideoCrafter2.
- Crowdsourced Human Annotation: Using a custom web interface, human annotators tracked down artifacts. Instead of just giving a thumbs-down, annotators drew spatio-temporal bounding boxes across video frames, selected an artifact category (e.g., Geometry, Motion, Physics), and wrote descriptive text explaining the issue.
- Automated Artifact Detection Baseline: To prove the dataset's utility, the authors built a dual-stage machine learning pipeline. It uses an Artifact Detector (based on RT-DETR) enhanced by a Temporal Fusion Module (which applies self-attention across 5 consecutive frames) to locate glitches. An Artifact Caption Generator (based on the GIT architecture) then generates a human-readable description of the localized bug.
Datasets
The primary contribution is the GeneVA (Generative Text to Video Artifacts) dataset. Compared to existing benchmarks like VBench or VideoScore, GeneVA is uniquely positioned as the only dataset providing both localized bounding boxes and descriptive text annotations.
- Total Prompts: 5,452
- Total Videos: 16,356 (amounting to 52,326 seconds of content)
- Total Human Annotations: 16,451
Key Insights and Ideas
- Sora Leads the Pack, But Still Glitches: OpenAI's Sora significantly outperformed Pika and VideoCrafter2 in overall visual quality and prompt alignment. However, it still frequently exhibited complex physical and structural artifacts.
- Semantic Mismatches Breed Artifacts: When a model fails to understand a prompt (a semantic mismatch), the frequency of physical and spatial artifacts spikes dramatically.
- The "Size" of a Glitch Matters, But Not Entirely: The authors found a statistically significant inverse correlation between bounding box coverage (the percentage of pixels/frames flagged as an artifact) and human quality ratings. However, the correlation is weak, suggesting that a tiny, well-placed artifact (like a deformed face) can ruin a video just as much as a massive geometry glitch.
- Temporal Context is Essential: In automated detection, looking at a single frame isn't enough. Incorporating temporal context via multi-frame fusion significantly boosts a model's ability to correctly identify video-specific artifacts.
Key Results
- Model Performance Comparison: Sora achieved notably higher user-submitted quality and prompt-alignment scores compared to Pika and VideoCrafter2, which shared similar, lower-rated score distributions.
- Artifact Detection Baseline: The proposed artifact detector achieved an Average Precision ($AP_{0.25}$) of 13.2% on the held-out test set. Removing the Temporal Fusion Module dropped performance to 10.3%, underscoring the importance of time-domain context.
- Zero-Shot Generalization: When evaluated on videos from Google's Veo—a state-of-the-art model completely unseen during training—the detector successfully localized and described novel artifacts (e.g., identifying that "people are not walking" or a "pizza moves on its own"), proving the dataset helps models learn generalized representations of visual errors.
Limitations and Future Work
The authors acknowledge a major budget-driven trade-off: to maximize prompt diversity, they did not collect repeated annotations for the same video. This means the dataset cannot currently be used to analyze inter-observer variability or establish a probabilistic consensus on how "noticeable" a given artifact is. Additionally, demographic data of the crowd-sourced annotators was omitted for privacy reasons.
For future work, the authors suggest leveraging GeneVA to train automated video inpainting models that can automatically "heal" localized artifacts, paving the way for cleaner, more reliable generative video tools.
Summary updated