VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary

Kevin Qinghong Lin, Mike Zheng Shou

Introduction

Current Video Large Language Models (VideoLLMs) typically operate by adapting text-based models to the visual domain. A significant bottleneck in this approach is the reliance on "subword vocabularies"—breaking words into fragments like 'happ' and 'y'. While effective for text, these fragments lack visual grounding and require computationally expensive token-by-token generation, making real-time video analysis nearly impossible for long-form content.

Comparison of traditional subword-based LLM vocabularies versus VLog's narration-based vocabulary and the resulting processing speedup.

VLog addresses this by shifting the paradigm from generation to "generative retrieval." Instead of predicting fragments, VLog treats human activities as a discrete vocabulary of complete narration phrases (e.g., "cutting a tomato"). This approach allows the model to reason about video content and retrieve an entire descriptive phrase in a single step, marrying the reasoning power of an LLM with the speed of a search engine.

Methodology

VLog's architecture is built on three primary pillars designed to optimize both speed and temporal reasoning.

1. Generative Retrieval

Unlike standard retrieval models that merely look for visual similarity, VLog uses a language model to process video frames and a query (e.g., "What happens next?"). The model generates a "retrieval token"—a single embedding representing the predicted outcome. This embedding is then matched against a pre-computed bank of narration embeddings using SigLIP. This allows the model to model causal relationships and predict future actions rather than just describing the current frame.

Comparison of (a) Generative, (b) Retrieval, and (c) Generative Retrieval (VLog) architectures, highlighting the pros and cons of each.

2. Narration Pair Encoding (NPE) and Hierarchical Indexing

To scale to millions of possible narrations, VLog employs Narration Pair Encoding (NPE). This technique decomposes complex descriptions into a Prefix (action) and a Postfix (detail), such as "Cut a potato" and "by left hand." These are organized into a hierarchy: Scenario → Prefix → Postfix. This structure enables the model to prune the search space effectively, making the search 15x faster than brute-force methods while maintaining accuracy.

3. Vocabulary Upgrading

To handle "Out-of-Vocabulary" (OOV) events, VLog includes an agentic workflow. When retrieval confidence is low, the system triggers a larger Multimodal Model (like LLaVA) to generate a description, which is then processed by an LLM to suggest and add new entries to the vocabulary on the fly.

Experimental Results

VLog was evaluated across several challenging benchmarks, including Ego4D, COIN, and the long-form QA benchmark EgoSchema. On EgoSchema, VLog achieved 43.1% accuracy, surpassing the comparable LLoVi model (38.8%) while being significantly more efficient.

The most striking result is the efficiency gain: VLog achieves a 10x to 20x speedup over traditional generative VideoLLMs. It can process a clip and retrieve a narration in approximately 0.018 seconds, compared to 0.362 seconds for token-by-token generation. Furthermore, the hierarchical indexing strategy proved to be 15x faster than a brute-force search of the 0.8M entry vocabulary.

Conclusion

By treating human activities as a language of discrete events rather than a sequence of subwords, VLog bridges the gap between high-level reasoning and real-time performance. Its generative retrieval mechanism offers a scalable, interpretable, and highly efficient path forward for video-language understanding, particularly for long-form and real-time applications like AR assistants.

Summary updated

Abstract

Human daily activities can be concisely narrated as sequences of routine events (e.g., turning off an alarm) in video streams, forming an event vocabulary. Motivated by this, we introduce VLog, a novel video understanding framework that define video narrations as vocabulary, going beyond the typical subword vocabularies in existing generative video-language models. Built on the lightweight language model GPT-2, VLog feature three key innovations: (i) A generative retrieval model, marrying language model's complex reasoning capabilities with contrastive retrieval's flexible upgrading over narration vocabulary. (ii) A hierarchical vocabulary derived from large-scale video narrations using our narration pair encoding algorithm, enabling efficient indexing of specific events (e.g., cutting a tomato) by identifying broader scenarios (e.g., kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary update strategy leveraging generative models to extend the vocabulary for novel events encountered during inference. To validate our approach, we introduce VidCap-Eval, a development set requiring concise narrations with reasoning relationships (e.g., before and after). Experiments on EgoSchema, COIN, and HiREST further demonstrate the effectiveness of VLog, highlighting its ability to generate concise, contextually accurate, and efficient narrations, offering a novel perspective on video understanding. Codes are released at https://github.com/showlab/VLog.
BibTeX
@inproceedings{lin2025vlog,
  author = {Kevin Qinghong Lin and Mike Zheng Shou},
  title = {{VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00306},
}
Figure