LAVIB: A Large-scale Video Interpolation Benchmark

Alex StergiouNeurIPS 2024

TL;DR

LAVIB (LArge-scale Video Interpolation Benchmark) is a massive new dataset designed to push the boundaries of Video Frame Interpolation (VFI). Featuring over 283,000 high-resolution 4K clips, it moves beyond simple scale by categorizing videos based on motion, sharpness, and lighting, providing a rigorous "in-the-wild" testing ground for model generalizability.

The Bottleneck in Video Frame Interpolation

Video Frame Interpolation (VFI)—the task of generating intermediate frames between existing ones—is essential for applications ranging from creating smooth slow-motion to reducing streaming bandwidth. However, the field's progress has been heavily bottlenecked by its training data.

Existing benchmarks like Vimeo-90K or UCF101 are often low-resolution, small in scale, or sourced from professional equipment under "perfect" conditions. Consequently, models trained on these datasets frequently struggle with real-world, "in-the-wild" footage that features extreme motion blur, poor lighting, or high-contrast scenes. Furthermore, the field has lacked standardized "stress tests" to evaluate how models handle out-of-distribution (OOD) scenarios—for example, a model trained exclusively on slow-moving scenes being deployed on fast-action sports.

Methodology: Building LAVIB

To address these shortcomings, the author developed an automated pipeline to source and curate diverse 4K/60fps footage from YouTube. The core innovation of LAVIB lies in its metric-driven selection process.

LAVIB dataset diversity visualized across four key metrics: motion, sharpness, contrast, and luminance.

The pipeline operates through several key steps:

  1. Automated Crawling: The system uses a diverse vocabulary of search terms (locations, activities, weather) to ensure a wide variety of scenes.
  2. Metric Computation: For every clip, the pipeline calculates four key attributes to quantify its visual characteristics:
  3. Average Flow Magnitude (AFM): Quantifies the intensity of motion.
  4. Average Laplacian Variance (ALV): Measures frame sharpness.
  5. Average Root Mean Square (ARMS): Measures image contrast.
  6. Average Relevant Luminance (ARL): Relates to overall brightness.
  7. Temporal & Spatial Cropping: To make 4K video manageable for GPU training, the pipeline extracts 1-second "tubelets" (spatial crops). It specifically selects the top 20% of high-motion segments and balances the final set across the other three metrics to avoid oversampling "easy," average cases.

Out-of-Distribution (OOD) Challenges

To rigorously test model robustness, the author intentionally created "unbalanced" splits within the dataset.

Out-of-distribution (OOD) challenge splits, illustrating the contrast between dark and bright video sets.

By establishing a training set of only dark videos and a test set of only bright videos (or low-motion vs. high-motion), LAVIB provides a standardized way to evaluate how well VFI models generalize across severe domain shifts.

Dataset Overview & Key Insights

The primary contribution, LAVIB, is a massive leap forward in scale and quality. It consists of 283,484 video segments (approximately 77.6 hours of footage) sourced from 17,204 ultra-HD (4K) clips, all standardized at 60 frames-per-second. This makes LAVIB roughly three times larger than Vimeo-90K in terms of video count, while offering significantly higher resolution (4K vs 720p).

Key insights from the benchmark include:

  • Diversity Over Density: Simply having more data isn't enough. By explicitly balancing for challenging attributes (like low contrast or high motion), LAVIB forces models to learn more robust features than they would from random, uncurated web-scraped data.
  • The "Rising Tide" of Pre-training: Pre-training on LAVIB improves performance even on smaller, specialized datasets. It acts as a foundational dataset that helps models understand general video structures before fine-tuning.
  • Quantifying the Gap: The OOD challenges provide a sobering look at current VFI limitations. Models trained on low-motion videos see a massive performance drop when tested on high-motion scenes, highlighting that current architectures remain highly sensitive to their training distributions.

Key Results

Using the FLAVR architecture as a baseline, the impact of LAVIB is clear:

  • Performance Boost: Models pre-trained on LAVIB achieved a PSNR of 33.44 on the test set, outperforming those trained on Vimeo-90K (32.10).
  • Cross-Dataset Generalization: A model pre-trained on LAVIB and then tested on Vimeo-90K improved PSNR by +0.43 dB compared to a model trained only on Vimeo-90K.
  • OOD Sensitivity: In the motion challenge, models moved from low-motion training to high-motion testing saw a PSNR drop of -2.64, identifying a clear "failure mode" for current VFI technology.

Limitations and Future Work

While LAVIB sets a new standard, the author acknowledges certain limitations. The 20% motion threshold and other pipeline steps were determined empirically; future work could involve more dynamic or learned selection criteria. Additionally, expanding the finite search query vocabulary could further increase the diversity of the "in-the-wild" scenes.

Finally, the author suggests that LAVIB’s standardized high-frame-rate data makes it an ideal playground for future research into frame-level quantization and knowledge distillation—specifically, training smaller, faster "student" models from large "teacher" models to make high-quality VFI more accessible.

Summary updated

BibTeX
@inproceedings{stergiou2024lavib,
  author = {Alex Stergiou},
  title = {{LAVIB: A Large-scale Video Interpolation Benchmark}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure