Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model

Keunwoo Peter Yu, Achal Dave, Rares Ambrus, Jean MercatarXiv 2024 · DOI · arXiv

Introduction

As Vision-Language Models (VLMs) transition from static image analysis to video understanding, they encounter a significant scaling hurdle known as "token explosion." A standard Vision Transformer (ViT) decomposes a single frame into hundreds of tokens. When processing long-form video with hundreds of frames, the resulting sequence can reach tens of thousands of tokens, quickly exceeding the memory and computational capacity of the Large Language Model (LLM) backbone.

General VLM architecture showing the flow from video frames through a ViT and Projector to an LLM.

The Projector Bottleneck

To manage this complexity, researchers use "projectors" to bridge the gap between the vision encoder and the LLM. Existing solutions typically fall into two categories:

  1. Pooling-based projectors: These reduce token counts by a linear factor (e.g., merging adjacent tokens). While effective for short clips, the token count still grows linearly with video length.
  2. Fixed-length projectors: Architectures like Flamingo's Perceiver Resampler compress video into a constant number of tokens regardless of duration. While efficient, these often "smear" temporal details, struggling with complex reasoning.

The Espresso Projector: Disentangled Compression

The authors of Espresso propose that fixed-length compression is viable if the architecture explicitly accounts for the distinct redundancies in spatial and temporal information. Espresso introduces a Dual-Path Processing strategy:

  • Spatial Path: A temporal pooler aggregates features across time for each patch, which is then compressed into $p$ tokens via a Q-Former. This captures the static layout and scene geometry.
  • Temporal Path: A spatial pooler aggregates features across patches for each frame, compressed into $t$ tokens. This focuses on motion and dynamic changes.
  • Segment-wise Processing: To scale to arbitrarily long videos, Espresso divides the input into $n$ segments, each compressed into a fixed $(p+t)$ budget. This maintains local temporal structure while keeping the total token count manageable.

By keeping the ViT and LLM (a 7B LLaVA-based model) frozen and training only the projector, Espresso remains highly efficient to train.

Experimental Results

Espresso was evaluated on diverse benchmarks, including short-form (MSVD-QA, MSRVTT-QA) and long-form (EgoSchema, MLVU) tasks.

Long-Form Performance

On the EgoSchema benchmark, Espresso (using 128 frames) achieved 40.17% accuracy, significantly outperforming pooling-based methods like STC (23.10%) and standard Perceiver-style approaches (36.08%).

The "Needle-in-a-Haystack" Challenge

To test long-range retrieval, the authors introduced NH-EgoSchema. A target video is hidden within three distractor clips, creating a 12-minute composite. Espresso maintained a robust performance of 32.98%, whereas MLP-based models like Tarsier failed due to Out-of-Memory (OOM) errors.

Key Insights

  • Disentanglement: Separating spatial and temporal paths provides "pre-digested" features to the LLM, reducing its cognitive load in distinguishing static objects from actions.
  • Spatial Redundancy: Aggressive spatial compression often improves performance, suggesting that LLMs can be distracted by redundant background details in long videos.
  • Efficiency: While the Espresso projector has more parameters than simple pooling layers (726M vs 50M), it is faster during inference because it drastically reduces the number of tokens processed by the LLM—the primary bottleneck.

Conclusion

Espresso demonstrates that intelligent, disentangled compression is the key to unlocking long-form video understanding in VLMs. By solving the token explosion problem without sacrificing temporal resolution, it provides a scalable path for real-time and streaming video applications.

Summary updated

Abstract

Recent advances in vision-language models (VLMs) have shown great promise in connecting images and text, but extending these models to long videos remains challenging due to the rapid growth in token counts. Models that compress videos by local aggregation in time or space have become popular for handling long-form inputs; however, these pooling-based projectors sacrifice the benefits of fixed-length representations that are crucial for streaming and efficient video understanding. We introduce $\texttt{Espresso}$, a new architecture that separately compresses spatial and temporal features into fixed-length sequences. $\texttt{Espresso}$ enables efficient video encoding while maintaining strong long-form reasoning capabilities. Experiments show that fixed-length compression combined with segment-wise processing offers a scalable and competitive alternative to pooling-based approaches. Our results demonstrate that fixed-length projectors, when properly designed and trained, remain a viable foundation for video-language modeling.
BibTeX
@misc{yu2024espresso,
  author = {Keunwoo Peter Yu and Achal Dave and Rares Ambrus and Jean Mercat},
  title = {{Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model}},
  year = {2024},
  eprint = {2412.04729},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.04729},
}
Figure