Introduction
Current Video Large Language Models (VideoLLMs) typically operate by adapting text-based models to the visual domain. A significant bottleneck in this approach is the reliance on "subword vocabularies"—breaking words into fragments like 'happ' and 'y'. While effective for text, these fragments lack visual grounding and require computationally expensive token-by-token generation, making real-time video analysis nearly impossible for long-form content.
VLog addresses this by shifting the paradigm from generation to "generative retrieval." Instead of predicting fragments, VLog treats human activities as a discrete vocabulary of complete narration phrases (e.g., "cutting a tomato"). This approach allows the model to reason about video content and retrieve an entire descriptive phrase in a single step, marrying the reasoning power of an LLM with the speed of a search engine.
Methodology
VLog's architecture is built on three primary pillars designed to optimize both speed and temporal reasoning.
1. Generative Retrieval
Unlike standard retrieval models that merely look for visual similarity, VLog uses a language model to process video frames and a query (e.g., "What happens next?"). The model generates a "retrieval token"—a single embedding representing the predicted outcome. This embedding is then matched against a pre-computed bank of narration embeddings using SigLIP. This allows the model to model causal relationships and predict future actions rather than just describing the current frame.
2. Narration Pair Encoding (NPE) and Hierarchical Indexing
To scale to millions of possible narrations, VLog employs Narration Pair Encoding (NPE). This technique decomposes complex descriptions into a Prefix (action) and a Postfix (detail), such as "Cut a potato" and "by left hand." These are organized into a hierarchy: Scenario → Prefix → Postfix. This structure enables the model to prune the search space effectively, making the search 15x faster than brute-force methods while maintaining accuracy.
3. Vocabulary Upgrading
To handle "Out-of-Vocabulary" (OOV) events, VLog includes an agentic workflow. When retrieval confidence is low, the system triggers a larger Multimodal Model (like LLaVA) to generate a description, which is then processed by an LLM to suggest and add new entries to the vocabulary on the fly.
Experimental Results
VLog was evaluated across several challenging benchmarks, including Ego4D, COIN, and the long-form QA benchmark EgoSchema. On EgoSchema, VLog achieved 43.1% accuracy, surpassing the comparable LLoVi model (38.8%) while being significantly more efficient.
The most striking result is the efficiency gain: VLog achieves a 10x to 20x speedup over traditional generative VideoLLMs. It can process a clip and retrieve a narration in approximately 0.018 seconds, compared to 0.362 seconds for token-by-token generation. Furthermore, the hierarchical indexing strategy proved to be 15x faster than a brute-force search of the 0.8M entry vocabulary.
Conclusion
By treating human activities as a language of discrete events rather than a sequence of subwords, VLog bridges the gap between high-level reasoning and real-time performance. Its generative retrieval mechanism offers a scalable, interpretable, and highly efficient path forward for video-language understanding, particularly for long-form and real-time applications like AR assistants.
Summary updated