TL;DR
This paper introduces "Focus," a novel hardware-software co-designed architecture for accelerating Vision-Language Models (VLMs). By employing a three-level "streaming concentration" strategy to efficiently eliminate redundancy at semantic, block, and vector granularities, Focus achieves significant speedups and energy reductions without sacrificing accuracy, making real-time VLM deployment more feasible.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) are becoming increasingly powerful, excelling at tasks like video captioning and visual question answering. However, their massive scale and the nature of video inputs (which contain a lot of redundant information) lead to substantial computational and memory overhead. This makes real-time deployment, especially on resource-constrained edge devices, a significant challenge. Existing methods for reducing this overhead, like token pruning or merging, often operate at a coarse, global level, leading to high runtime costs and poor hardware efficiency. The authors aim to bridge this gap by proposing a more integrated, fine-grained approach that works hand-in-hand with hardware accelerators.
Methods
The core innovation of Focus is its multilevel streaming concentration approach, which progressively eliminates redundancy:
- Semantic-Guided Token Pruning: Unlike previous methods that rely on static metrics, Focus uses cross-modal attention to identify and retain only the visual tokens that are semantically relevant to the given text prompt. This is done by a "Semantic Concentrator" (SEC) module that analyzes attention maps.
- Spatial-Temporal Block-Level Concentration: After semantic pruning, Focus groups the remaining tokens into small spatial-temporal blocks. It then uses a "Similarity Concentrator" (SIC) module to perform localized comparisons within these blocks, identifying and merging similar tokens. This is inspired by convolutional operations and avoids costly global comparisons.
- Vector-Level Redundancy Removal: At the finest granularity, Focus operates on vectors within token embeddings. It uses a "motion-aware matching" technique to capture subtle similarities and redundancies that might be missed by token-level comparisons. This is crucial for handling variations like motion or slight object shifts between frames.
These concentration steps are tightly integrated with a systolic-array accelerator architecture. Key architectural features include:
- GEMM Tiling: Standard technique for efficient matrix multiplication, which Focus leverages for its block-level operations.
- Convolution-Style Layout: A novel data reordering to facilitate block-wise comparisons within memory.
- Streaming-Friendly Design: All concentration steps are designed to operate on-chip and in a streaming fashion, minimizing off-chip memory access and latency.
Datasets
The paper evaluates Focus using several representative VLMs on established benchmarks:
- Training/Evaluation Models: LLaVA-OneVision-7B, LLaVA-Video-7B, MiniCPM-V-2.6, and Qwen2.5-VL.
- Benchmarks: VideoMME (VMME), MVBench (MVB), and MLVU for video understanding tasks. VQAv2, MME, and MMBench were also used for image-based VLM evaluations.
Key Insights and Ideas
- Prompt-Dependent Importance: The importance of visual information in VLMs is not static but heavily depends on the text prompt. Static pruning methods miss this crucial aspect, whereas Focus's semantic pruning leverages cross-modal attention to adapt to the prompt.
- Fine-Grained Redundancy is Key: Simply looking at whole tokens is too coarse. Significant redundancy exists at the vector level within token embeddings, especially in video data. Focusing on vector-wise similarity unlocks much higher sparsity without accuracy loss.
- Hardware-Algorithm Co-Design Pays Off: By designing the concentration algorithms (semantic pruning, block/vector similarity) to align perfectly with hardware primitives like GEMM tiling and streaming dataflow, Focus achieves unprecedented efficiency. This is a departure from purely algorithmic approaches or hardware adaptations of existing algorithms.
- On-Chip Processing is Paramount: Moving redundancy elimination from off-chip (like video codecs) to on-chip processing dramatically reduces memory bandwidth requirements and latency.
Key Results
- Speedup: Focus achieves up to 2.4x speedup compared to state-of-the-art accelerators and an average of 4.47x speedup over a vanilla systolic array.
- Energy Efficiency: Focus demonstrates a 3.3x reduction in energy compared to state-of-the-art accelerators and an average of 4.67x improvement over a vanilla systolic array.
- Sparsity: Achieves an average computational sparsity of 80.2%, significantly outperforming baselines like AdapTiV (40-50%) and CMC (40-50%).
- Accuracy: Maintains high accuracy, with an average degradation of only 1.20% compared to uncompressed models.
- Area Overhead: The Focus unit adds only 2.7% to the overall area of a systolic-array accelerator.
Limitations and Future Work
- Worst-Case Scenario: In the absolute worst case (no similarity), sparsity drops, and latency increases. However, the paper shows this is rare and the system remains robust.
- Design Space Trade-offs: The paper explores design space trade-offs for parameters like GEMM tile size and vector size, identifying optimal points but acknowledging that these can influence latency and buffer usage.
- Future Work: The authors suggest extending Focus to image-based VLMs (by treating a single image as a one-frame video) and to Vision-Language-Action (VLA) models, leveraging its ability to eliminate redundant information. They also mention exploring dynamic adaptation of pruning ratios based on post-softmax attention thresholds or top-p pruning.
Summary updated