VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges

Yuxuan Wang, Cihang Xie, Yang Liu, Zilong Zheng

Abstract

Recent advancements in large-scale video-language models have shown significant potential for real-time planning and detailed interactions. However, their high computational demands and the scarcity of annotated datasets limit their practicality for academic researchers. In this work, we introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens within bridge layers to allow for the encoding of entire video sequences alongside historical visual data, effectively preserving semantic continuity and enhancing model performance across various tasks. This approach includes recurrent memory tokens and a SceneTilling algorithm, which segments videos into independent semantic units to preserve semantic integrity. Empirically, VideoLLaMB significantly outstrips existing video-language models, demonstrating a 5.5 points improvement over its competitors across three VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive results on the MVBench show that VideoLLaMB-7B achieves markedly better results than previous 7B models of same LLM. Remarkably, it maintains robust performance as PLLaVA even as video length increases up to 8 times. Besides, the frame retrieval results on our specialized Needle in a Video Haystack (NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately identifying specific frames within lengthy videos. Our SceneTilling algorithm also enables the generation of streaming video captions directly, without necessitating additional training. In terms of efficiency, VideoLLaMB, trained on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear GPU memory scaling, ensuring both high performance and cost-effectiveness, thereby setting a new foundation for long-form video-language models in both academic and practical applications.

Introduction

Understanding long, streaming video is a massive challenge for current Large Multimodal Models (LMMs). Most existing models rely on heavy compression—such as aggressive downsampling or pooling—which often discards the fine-grained visual cues necessary for detailed reasoning. Alternatively, models that segment video into short, arbitrary clips often lose the "semantic flow," failing to connect actions at the beginning of a video to those at the end.

Furthermore, training foundational video-language models is prohibitively expensive for most academic researchers. VideoLLaMB addresses these challenges by introducing a framework designed for "long-term memory" in video understanding, achieving state-of-the-art results on long-form benchmarks while remaining efficient enough for standard hardware.

Methodology

VideoLLaMB utilizes three core components to maintain semantic continuity across long sequences:

  1. SceneTiling: Instead of cutting a video into arbitrary time-based chunks, this model-free algorithm uses visual features (specifically [CLS] tokens from a ViT) to calculate similarity between adjacent frames. It identifies natural semantic breaks, dividing the video into coherent "scenes."
  2. Recurrent Memory Bridge Layers: The model prepends a set of "memory tokens" to the visual features of each scene. These tokens pass through a Transformer-based bridge layer, where they absorb the scene's information. The updated tokens are then passed to the next scene, creating a recursive chain of information.
  3. Memory Cache with Retrieval: To prevent gradient vanishing and forgetting, the system maintains a cache of all previous memory tokens. At each step, a cross-attention mechanism retrieves relevant history from the cache to update the model's current state.
VideoLLaMB Architecture Overview

Crucially, this bridge architecture sits between a frozen visual encoder and a frozen LLM, significantly reducing the need for expensive retraining while allowing the model to process hundreds of frames.

Experimental Results

The researchers evaluated VideoLLaMB across a diverse battery of benchmarks, including EgoSchema, NExTQA, VideoMME, and MVBench. The model also underwent a new stress test: Needle In A Video Haystack (NIAVH), which requires the model to find a 1-second "needle" (text, image, or video clip) within a 320-second "haystack."

Key Performance Metrics

  • VideoQA: VideoLLaMB outperformed existing 7B-parameter models by 4.2 points on average across four major benchmarks.
  • Embodied Planning: On the EgoPlan benchmark, it achieved a 2.06 point improvement over the previous leader, PLLaVA.
  • Needle Retrieval: In the NIAVH test, VideoLLaMB significantly outperformed competitors like MA-LMM and LongVA in identifying specific details buried deep within long sequences.

Efficiency and Scalability

A primary advantage of VideoLLaMB is its linear scaling. By using a recurrent bridge rather than full self-attention over every frame, the model’s GPU memory usage remains manageable even as video length increases. This allows the processing of 320 frames on a single Nvidia A100—a task that typically requires massive server clusters for competing architectures.

Memory Usage Scaling Comparison

Additionally, because the memory bridge is recurrent, a model trained on only 16 frames can extrapolate to videos up to 8x that length during inference without a collapse in performance. The model achieved an average inference time of 4.21 seconds on 300-second videos, nearly twice as fast as some competing architectures.

Conclusion

VideoLLaMB demonstrates that semantic continuity and linear efficiency are achievable for long-context video understanding. By combining SceneTiling with recurrent memory bridges, the framework provides a robust foundation for both academic research and practical applications in video reasoning and planning. Future work will focus on deeper integration between the bridge memory and the LLM’s internal KV cache to further enhance long-term retention and reasoning efficiency.

Summary updated

Abstract

Recent advancements in large-scale video-language models have shown significant potential for real-time planning and detailed interactions. However, their high computational demands and the scarcity of annotated datasets limit their practicality for academic researchers. In this work, we introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens within bridge layers to allow for the encoding of entire video sequences alongside historical visual data, effectively preserving semantic continuity and enhancing model performance across various tasks. This approach includes recurrent memory tokens and a SceneTilling algorithm, which segments videos into independent semantic units to preserve semantic integrity. Empirically, VideoLLaMB significantly outstrips existing video-language models, demonstrating a 5.5 points improvement over its competitors across three VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive results on the MVBench show that VideoLLaMB-7B achieves markedly better results than previous 7B models of same LLM. Remarkably, it maintains robust performance as PLLaVA even as video length increases up to 8 times. Besides, the frame retrieval results on our specialized Needle in a Video Haystack (NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately identifying specific frames within lengthy videos. Our SceneTilling algorithm also enables the generation of streaming video captions directly, without necessitating additional training. In terms of efficiency, VideoLLaMB, trained on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear GPU memory scaling, ensuring both high performance and cost-effectiveness, thereby setting a new foundation for long-form video-language models in both academic and practical applications.
BibTeX
@misc{wang2024videollamb,
  author = {Yuxuan Wang and Cihang Xie and Yang Liu and Zilong Zheng},
  title = {{VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges}},
  year = {2024},
  eprint = {2409.01071},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01071},
}
Figure