GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifacts

Jenna Kang, Maria Silva, Patsorn Sangkloy, Kenneth Chen, Niall Williams, Qi Sun

TL;DR

The authors introduce GeneVA, a large-scale dataset containing over 16,000 AI-generated videos coupled with rich, human-annotated spatio-temporal artifact labels. Unlike previous benchmarks, GeneVA provides both fine-grained bounding boxes and free-form textual descriptions of video glitches, enabling the training of a baseline automated artifact detector that generalizes to unseen generative models.

What Problem Does This Paper Address?

While text-to-video (T2V) diffusion models have made staggering leaps in generating dynamic content, they frequently suffer from jarring visual bugs. These range from spatial defects (like distorted anatomy) to temporal anomalies (like objects randomly popping out of existence or violating basic laws of physics).

Standard automated metrics like Fréchet Video Distance (FVD) often correlate poorly with actual human perception of these glitches. To fix this, researchers need human-annotated data. However, existing video quality datasets only provide holistic, video-level scores. They don't specify where or when an artifact occurs, nor do they describe its nature, leaving a major gap in our ability to systematically diagnose and mitigate T2V errors.

Methods

The researchers built a comprehensive pipeline to collect, annotate, and model video artifacts:

  1. Prompt Selection & Generation: They sampled 5,452 diverse, real-world prompts from the VidProM dataset using kernel herding on text embeddings to avoid semantic bias. They then generated videos using three prominent T2V models: OpenAI's Sora, Pika, and VideoCrafter2.
  2. Crowdsourced Human Annotation: Using a custom web interface, human annotators tracked down artifacts. Instead of just giving a thumbs-down, annotators drew spatio-temporal bounding boxes across video frames, selected an artifact category (e.g., Geometry, Motion, Physics), and wrote descriptive text explaining the issue.
  3. Automated Artifact Detection Baseline: To prove the dataset's utility, the authors built a dual-stage machine learning pipeline. It uses an Artifact Detector (based on RT-DETR) enhanced by a Temporal Fusion Module (which applies self-attention across 5 consecutive frames) to locate glitches. An Artifact Caption Generator (based on the GIT architecture) then generates a human-readable description of the localized bug.
Overview of the video annotation interface where users select categories, describe artifacts, and draw bounding boxes.

Datasets

The primary contribution is the GeneVA (Generative Text to Video Artifacts) dataset. Compared to existing benchmarks like VBench or VideoScore, GeneVA is uniquely positioned as the only dataset providing both localized bounding boxes and descriptive text annotations.

  • Total Prompts: 5,452
  • Total Videos: 16,356 (amounting to 52,326 seconds of content)
  • Total Human Annotations: 16,451

Key Insights and Ideas

  • Sora Leads the Pack, But Still Glitches: OpenAI's Sora significantly outperformed Pika and VideoCrafter2 in overall visual quality and prompt alignment. However, it still frequently exhibited complex physical and structural artifacts.
  • Semantic Mismatches Breed Artifacts: When a model fails to understand a prompt (a semantic mismatch), the frequency of physical and spatial artifacts spikes dramatically.
  • The "Size" of a Glitch Matters, But Not Entirely: The authors found a statistically significant inverse correlation between bounding box coverage (the percentage of pixels/frames flagged as an artifact) and human quality ratings. However, the correlation is weak, suggesting that a tiny, well-placed artifact (like a deformed face) can ruin a video just as much as a massive geometry glitch.
  • Temporal Context is Essential: In automated detection, looking at a single frame isn't enough. Incorporating temporal context via multi-frame fusion significantly boosts a model's ability to correctly identify video-specific artifacts.
Figure 1. Human annotators were asked to describe, in a free text form, artifacts in AI-generated video. A word cloud of the most frequent artifacts mentioned by the annotators is shown here.

Key Results

  • Model Performance Comparison: Sora achieved notably higher user-submitted quality and prompt-alignment scores compared to Pika and VideoCrafter2, which shared similar, lower-rated score distributions.
  • Artifact Detection Baseline: The proposed artifact detector achieved an Average Precision ($AP_{0.25}$) of 13.2% on the held-out test set. Removing the Temporal Fusion Module dropped performance to 10.3%, underscoring the importance of time-domain context.
  • Zero-Shot Generalization: When evaluated on videos from Google's Veo—a state-of-the-art model completely unseen during training—the detector successfully localized and described novel artifacts (e.g., identifying that "people are not walking" or a "pizza moves on its own"), proving the dataset helps models learn generalized representations of visual errors.
Box plots illustrating the inverse relationship between spatio-temporal bounding box coverage and human quality ratings across different models.

Limitations and Future Work

The authors acknowledge a major budget-driven trade-off: to maximize prompt diversity, they did not collect repeated annotations for the same video. This means the dataset cannot currently be used to analyze inter-observer variability or establish a probabilistic consensus on how "noticeable" a given artifact is. Additionally, demographic data of the crowd-sourced annotators was omitted for privacy reasons.

For future work, the authors suggest leveraging GeneVA to train automated video inpainting models that can automatically "heal" localized artifacts, paving the way for cleaner, more reliable generative video tools.

Summary updated

Abstract

Recent advances in probabilistic generative models have extended capabilities from static image synthesis to text-driven video generation. However, the inherent randomness of their generation process can lead to unpredictable artifacts, such as impossible physics and temporal inconsistency. Progress in addressing these challenges requires systematic benchmarks, yet existing datasets primarily focus on generative images due to the unique spatio-temporal complexities of videos. To bridge this gap, we introduce GeneVA, a large-scale artifact dataset with rich human annotations that focuses on spatio-temporal artifacts in videos generated from natural text prompts. We hope GeneVA can enable and assist critical applications, such as benchmarking model performance and improving generative video quality.
BibTeX
@inproceedings{kang2026geneva,
  author = {Jenna Kang and Maria Silva and Patsorn Sangkloy and Kenneth Chen and Niall Williams and Qi Sun},
  title = {{GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifacts}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
}
Figure