Automated Scaling of Video Language Datasets

This direction is split into other directions: manual annotation bottleneck (annotation scalability bottleneck). Its last version stays readable and citable here.

Version 1 of 1217 papers217 conceptsfrozen 2026-10-04

In the evolution of computer vision, video has always been the "expensive" sibling of the static image. While a single image can be labeled in seconds, a video requires an annotator to watch, track, and reason across time—a process that Kristen Grauman and colleagues note can take ten times the duration of the footage itself for tasks like de-identification. This "video annotation bottleneck" has historically limited the scale of video-language datasets, preventing them from reaching the billions of samples seen in image-text counterparts.

To overcome this, researchers are shifting away from manual labor toward automated, LLM-driven pipelines that synthesize, clean, and reason about video data at scale.

The LLM as a Scalable Annotator

The most significant shift in recent work is the use of Large Language Models (LLMs) to bridge the gap between raw video metadata and high-level reasoning. Instead of asking humans to write questions, researchers like Ruchit Rawal in CinePile leverage movie audio descriptions and LLMs to generate hundreds of thousands of multiple-choice questions.

The intuition is simple: if you have a high-quality text description of a video (like a screenplay or a professional narration), an LLM can "imagine" the visual scene and generate complex, counterfactual, or causal questions that would be prohibitively expensive for a human to draft. This approach was taken further by Zhengqian Wu and the creators of FriendsQA, who developed the StoryMind multi-agent framework to automatically generate and filter 44,000 questions about long-range storylines in sitcoms.

Mining Weak Supervision from the Web

Before LLMs were powerful enough to act as annotators, researchers looked for "free" labels already present in the world. Antoine Miech pioneered this with HowTo100M, which used automatically transcribed narrations (ASR) from YouTube tutorials to create 136 million video-text pairs.

While ASR is often noisy and misaligned, the sheer volume of data allows models to learn robust representations. Other researchers have tapped into even more specialized "weak" signals:

  • DVS Tracks: Atousa Torabi utilized Descriptive Video Services (DVS)—narrations for the visually impaired—to create the M-VAD dataset.
  • Temporal Logic: Ziyun Zeng introduced the TVTS task, which teaches models the "logic of time" by forcing them to re-order shuffled ASR sentences based on video content.

Agentic Curation and Synthetic Pipelines

The latest frontier moves beyond static generation to agentic workflows—systems that can reason about, edit, and filter data iteratively. Shentong Mo introduced AVAgent, which uses an LLM-based agent to "fix" messy audio-visual data by removing noise and synchronizing tracks before training. Similarly, the VC-Agent system allows researchers to collect highly specific video datasets (e.g., "a cat lying down, but not a black one") by using an MLLM to iteratively filter web-crawled clips based on minimal human feedback.

When real-world data is too difficult to label, researchers turn to synthetic generation. Daoyuan Chen and colleagues built HumanVBench using synthetic pipelines to evaluate human-centric nuances like emotion and speech-visual alignment, ensuring "ground truth" accuracy that is often lost in manual human labeling.

From Recognition to Deep Reasoning

The result of these automated scaling efforts is a new generation of benchmarks that move from "What is in this frame?" to "Why did this happen?".

  • Causal Reasoning: CausalStep by Xuchen Li enforces a stepwise protocol where models must explain sequential cause-and-effect.
  • Fine-Grained Understanding: VideoVista and MVBench (by Kunchang Li) provide tens of thousands of tasks across dozens of categories, revealing that while models are getting better at identifying objects, they still struggle with the temporal "connect-the-dots" logic that defines human video understanding.

By automating the curation and annotation process, the field is finally moving toward the scale necessary to train foundation models that can truly "watch" and understand the world.

Papers in this version

Showing 60 of 217 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_scaling_of_video_language_data_8fbe65ce,
  title = {Automated Scaling of Video Language Datasets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-scaling-of-video-language-datasets/rdr_8fbe65cec90e4166b2507663ad7ec879},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_scaling_of_video_language_data_8fbe65ce_v1,
  title = {Automated Scaling of Video Language Datasets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-scaling-of-video-language-datasets/rdr_8fbe65cec90e4166b2507663ad7ec879/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}