Introduction
As Vision-Language Models (VLMs) transition from static image analysis to video understanding, they encounter a significant scaling hurdle known as "token explosion." A standard Vision Transformer (ViT) decomposes a single frame into hundreds of tokens. When processing long-form video with hundreds of frames, the resulting sequence can reach tens of thousands of tokens, quickly exceeding the memory and computational capacity of the Large Language Model (LLM) backbone.
The Projector Bottleneck
To manage this complexity, researchers use "projectors" to bridge the gap between the vision encoder and the LLM. Existing solutions typically fall into two categories:
- Pooling-based projectors: These reduce token counts by a linear factor (e.g., merging adjacent tokens). While effective for short clips, the token count still grows linearly with video length.
- Fixed-length projectors: Architectures like Flamingo's Perceiver Resampler compress video into a constant number of tokens regardless of duration. While efficient, these often "smear" temporal details, struggling with complex reasoning.
The Espresso Projector: Disentangled Compression
The authors of Espresso propose that fixed-length compression is viable if the architecture explicitly accounts for the distinct redundancies in spatial and temporal information. Espresso introduces a Dual-Path Processing strategy:
- Spatial Path: A temporal pooler aggregates features across time for each patch, which is then compressed into $p$ tokens via a Q-Former. This captures the static layout and scene geometry.
- Temporal Path: A spatial pooler aggregates features across patches for each frame, compressed into $t$ tokens. This focuses on motion and dynamic changes.
- Segment-wise Processing: To scale to arbitrarily long videos, Espresso divides the input into $n$ segments, each compressed into a fixed $(p+t)$ budget. This maintains local temporal structure while keeping the total token count manageable.
By keeping the ViT and LLM (a 7B LLaVA-based model) frozen and training only the projector, Espresso remains highly efficient to train.
Experimental Results
Espresso was evaluated on diverse benchmarks, including short-form (MSVD-QA, MSRVTT-QA) and long-form (EgoSchema, MLVU) tasks.
Long-Form Performance
On the EgoSchema benchmark, Espresso (using 128 frames) achieved 40.17% accuracy, significantly outperforming pooling-based methods like STC (23.10%) and standard Perceiver-style approaches (36.08%).
The "Needle-in-a-Haystack" Challenge
To test long-range retrieval, the authors introduced NH-EgoSchema. A target video is hidden within three distractor clips, creating a 12-minute composite. Espresso maintained a robust performance of 32.98%, whereas MLP-based models like Tarsier failed due to Out-of-Memory (OOM) errors.
Key Insights
- Disentanglement: Separating spatial and temporal paths provides "pre-digested" features to the LLM, reducing its cognitive load in distinguishing static objects from actions.
- Spatial Redundancy: Aggressive spatial compression often improves performance, suggesting that LLMs can be distracted by redundant background details in long videos.
- Efficiency: While the Espresso projector has more parameters than simple pooling layers (726M vs 50M), it is faster during inference because it drastically reduces the number of tokens processed by the LLM—the primary bottleneck.
Conclusion
Espresso demonstrates that intelligent, disentangled compression is the key to unlocking long-form video understanding in VLMs. By solving the token explosion problem without sacrificing temporal resolution, it provides a scalable path for real-time and streaming video applications.
Summary updated