Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models

Chiyue Wei, Christine C. Guo, Junyao Zhang, Haoxuan Shan, Yifan Xu, Ziyue Zhang, Yudong Liu, Qinsi Wang, Changchun Zhou, Hai Helen Li, Yuansi Chen

TL;DR

This paper introduces "Focus," a novel hardware-software co-designed architecture for accelerating Vision-Language Models (VLMs). By employing a three-level "streaming concentration" strategy to efficiently eliminate redundancy at semantic, block, and vector granularities, Focus achieves significant speedups and energy reductions without sacrificing accuracy, making real-time VLM deployment more feasible.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) are becoming increasingly powerful, excelling at tasks like video captioning and visual question answering. However, their massive scale and the nature of video inputs (which contain a lot of redundant information) lead to substantial computational and memory overhead. This makes real-time deployment, especially on resource-constrained edge devices, a significant challenge. Existing methods for reducing this overhead, like token pruning or merging, often operate at a coarse, global level, leading to high runtime costs and poor hardware efficiency. The authors aim to bridge this gap by proposing a more integrated, fine-grained approach that works hand-in-hand with hardware accelerators.

Methods

The core innovation of Focus is its multilevel streaming concentration approach, which progressively eliminates redundancy:

  1. Semantic-Guided Token Pruning: Unlike previous methods that rely on static metrics, Focus uses cross-modal attention to identify and retain only the visual tokens that are semantically relevant to the given text prompt. This is done by a "Semantic Concentrator" (SEC) module that analyzes attention maps.
  2. Spatial-Temporal Block-Level Concentration: After semantic pruning, Focus groups the remaining tokens into small spatial-temporal blocks. It then uses a "Similarity Concentrator" (SIC) module to perform localized comparisons within these blocks, identifying and merging similar tokens. This is inspired by convolutional operations and avoids costly global comparisons.
  3. Vector-Level Redundancy Removal: At the finest granularity, Focus operates on vectors within token embeddings. It uses a "motion-aware matching" technique to capture subtle similarities and redundancies that might be missed by token-level comparisons. This is crucial for handling variations like motion or slight object shifts between frames.

These concentration steps are tightly integrated with a systolic-array accelerator architecture. Key architectural features include:

  • GEMM Tiling: Standard technique for efficient matrix multiplication, which Focus leverages for its block-level operations.
  • Convolution-Style Layout: A novel data reordering to facilitate block-wise comparisons within memory.
  • Streaming-Friendly Design: All concentration steps are designed to operate on-chip and in a streaming fashion, minimizing off-chip memory access and latency.

Datasets

The paper evaluates Focus using several representative VLMs on established benchmarks:

  • Training/Evaluation Models: LLaVA-OneVision-7B, LLaVA-Video-7B, MiniCPM-V-2.6, and Qwen2.5-VL.
  • Benchmarks: VideoMME (VMME), MVBench (MVB), and MLVU for video understanding tasks. VQAv2, MME, and MMBench were also used for image-based VLM evaluations.

Key Insights and Ideas

  • Prompt-Dependent Importance: The importance of visual information in VLMs is not static but heavily depends on the text prompt. Static pruning methods miss this crucial aspect, whereas Focus's semantic pruning leverages cross-modal attention to adapt to the prompt.
  • Fine-Grained Redundancy is Key: Simply looking at whole tokens is too coarse. Significant redundancy exists at the vector level within token embeddings, especially in video data. Focusing on vector-wise similarity unlocks much higher sparsity without accuracy loss.
  • Hardware-Algorithm Co-Design Pays Off: By designing the concentration algorithms (semantic pruning, block/vector similarity) to align perfectly with hardware primitives like GEMM tiling and streaming dataflow, Focus achieves unprecedented efficiency. This is a departure from purely algorithmic approaches or hardware adaptations of existing algorithms.
  • On-Chip Processing is Paramount: Moving redundancy elimination from off-chip (like video codecs) to on-chip processing dramatically reduces memory bandwidth requirements and latency.

Key Results

  • Speedup: Focus achieves up to 2.4x speedup compared to state-of-the-art accelerators and an average of 4.47x speedup over a vanilla systolic array.
  • Energy Efficiency: Focus demonstrates a 3.3x reduction in energy compared to state-of-the-art accelerators and an average of 4.67x improvement over a vanilla systolic array.
  • Sparsity: Achieves an average computational sparsity of 80.2%, significantly outperforming baselines like AdapTiV (40-50%) and CMC (40-50%).
  • Accuracy: Maintains high accuracy, with an average degradation of only 1.20% compared to uncompressed models.
  • Area Overhead: The Focus unit adds only 2.7% to the overall area of a systolic-array accelerator.

Limitations and Future Work

  • Worst-Case Scenario: In the absolute worst case (no similarity), sparsity drops, and latency increases. However, the paper shows this is rare and the system remains robust.
  • Design Space Trade-offs: The paper explores design space trade-offs for parameters like GEMM tile size and vector size, identifying optimal points but acknowledging that these can influence latency and buffer usage.
  • Future Work: The authors suggest extending Focus to image-based VLMs (by treating a single image as a one-frame video) and to Vision-Language-Action (VLA) models, leveraging its ability to eliminate redundant information. They also mention exploring dynamic adaptation of pruning ratios based on post-softmax attention thresholds or top-p pruning.

Summary updated

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance on tasks such as video captioning and visual question answering. However, their growing scale and video-level inputs lead to significant computational and memory overhead, posing challenges for real-time deployment on hardware accelerators. While prior work attempts to reduce redundancy via token pruning or merging, these methods typically operate at coarse granularity and incur high runtime overhead due to global token-level operations. In this study, we propose Focus, a Streaming Concentration Architecture that efficiently accelerates VLM inference through progressive, fine-grained redundancy elimination. Focus introduces a multilevel concentration paradigm that hierarchically compresses vision-language inputs at three levels: (1) semantic-guided token pruning based on textual prompts, (2) spatial-temporal blocklevel concentration using localized comparisons, and (3) vectorlevel redundancy removal via motion-aware matching. All concentration steps are tightly co-designed with the architecture to support streaming-friendly, on-chip execution. Focus leverages GEMM tiling, convolution-style layout, and cross-modal attention to minimize off-chip access while enabling high throughput. Implemented as a modular unit within a systolic-array accelerator, Focus achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.4 \times$</tex> speedup and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.3 \times$</tex> reduction in energy, significantly outperforming state-of-the-art accelerator in both performance and energy efficiency. Full-stack implementation of Focus is open-sourced at https://github.com/dubcyfor3/Focus.
BibTeX
@misc{wei2025focus,
  author = {Chiyue Wei and Christine C. Guo and Junyao Zhang and Haoxuan Shan and Yifan Xu and Ziyue Zhang and Yudong Liu and Qinsi Wang and Changchun Zhou and Hai Helen Li and Yuansi Chen},
  title = {{Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models}},
  year = {2025},
  eprint = {2512.14661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.14661},
}
Figure