Scaling Video AI Through Image Transfer and Unlabeled Data

59 concepts58 papers1 proposal

The primary bottleneck in video-language modeling is a massive data disparity. While researchers can leverage billions of image-text pairs scraped from the web, high-quality video-text datasets are orders of magnitude smaller and significantly more expensive to produce. As noted by Rohit Girdhar et al. in their work on Emu Video, this scarcity makes it difficult for models to learn the complex temporal dynamics and diverse visual concepts required for realistic motion.

To overcome this "chronic scarcity," the field has shifted away from training on video-text pairs alone, instead developing strategies to "recycle" spatial knowledge from images and motion knowledge from unlabeled video.

Inheriting Spatial Intelligence from Images

The most successful approach to data scarcity is leveraging the "spatial intelligence" already captured by large-scale image models. Since image-text datasets like MS-COCO or LAION are vast, they provide a rich vocabulary of visual concepts that video-specific datasets lack.

Subhashini Venugopalan et al. demonstrated early on that transfer learning from image captioning could produce more fluid and diverse video descriptions than template-based methods. Modern generative models take this further through factorization. For instance, Emu Video simplifies the generation process by first creating a high-quality static image from text and then "animating" that image into a video. This allows the model to rely on the photorealistic standards of text-to-image models while using smaller video datasets only to learn motion.

Similarly, Wenyi Hong et al. designed CogVideo to "inherit" knowledge from pretrained image transformers. By using a dual-channel attention mechanism, the model maintains the spatial integrity learned from billions of images while adding temporal layers to handle frame-to-frame consistency.

Learning Motion from Text-Free Video

If image-text pairs provide the "what," unlabeled video provides the "how." Massive amounts of video exist without captions, and researchers have found ways to use this "silent" data to teach models about physics and motion.

The Make-A-Video framework by Uriel Singer et al. is a landmark example. It generates high-quality video without ever seeing a single paired text-video example during training. Instead, it learns world appearance from images and motion dynamics from unlabeled video footage. Xiang Wang et al. further validated this in TF-T2V, showing that simply adding more unlabeled video data significantly improves visual quality and motion consistency by decoupling spatial appearance from temporal dynamics.

Synthetic Supervision and LLM Narrators

Another strategy involves using Large Language Models (LLMs) to bridge the gap between raw video and descriptive text. Standard video benchmarks often suffer from limited vocabularies or "noisy" ASR transcripts where the speech doesn't match the visual action.

In LaViLa, Yue Zhao et al. used LLMs as "visually-conditioned narrators" to automatically generate dense, diverse descriptions for raw video. This creates a high-quality synthetic dataset that can outperform human-annotated data, which is often sparse. Similarly, the VISTA framework by Weiming Ren et al. addresses the lack of long-form data by "stitching" short clips into complex synthetic videos and using LLMs to generate corresponding instructions.

Efficient Adaptation and Context Transfer

Finally, researchers are finding that model architecture can sometimes compensate for data scarcity. Instead of training massive video models from scratch, lightweight adapters can be used to "plug" temporal capabilities into existing frozen models.

Ruyang Liu et al. introduced BT-Adapter, which allows image-language models to handle video conversations by training only a small parallel temporal branch. This process takes just hours, yet it achieves state-of-the-art results without expensive video instruction tuning.

Perhaps most surprisingly, Peiyuan Zhang et al. discovered in LongVA that models can "inherit" the ability to process long videos simply by extending the context window of their underlying language model. By training on text and short images alone, the model learned to process thousands of video frames, suggesting that the bottleneck for long-form understanding may be linguistic context rather than video-specific training data.

Key Foundational Work

Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014) — contributed 1 concept elements to this direction.
Taxonomic classification for web-based videos (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_video_text_data_scarcity_25cc1a16,
  title = {video-text data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/video-text-data-scarcity/rdr_25cc1a16132945559e19fda78715b000},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_video_text_data_scarcity_25cc1a16_v1,
  title = {video-text data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/video-text-data-scarcity/rdr_25cc1a16132945559e19fda78715b000/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake