Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models

Chiyue Wei, Cong Guo, Junyao Zhang, Haoxuan Shan, Yifan Xu, Ziyue Zhang, Yudong Liu, Qinsi Wang, Changchun Zhou, Hai "Helen" Li, Yiran Chen

TL;DR

Focus is a hardware-software co-designed architecture that accelerates Vision-Language Model (VLM) inference by progressively eliminating visual redundancy at three distinct levels: semantic, block, and vector. Operating entirely on-chip in a streaming fashion, Focus achieves a 2.4× speedup and a 3.3× reduction in energy consumption over state-of-the-art accelerators while maintaining high model accuracy.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) excel at multimodal tasks like video captioning and visual question answering, but their deployment on edge devices is severely bottlenecked by computational and memory overhead. In video-based VLMs, visual tokens make up a staggering 98% to 99% of the total input sequence. While video frames contain massive amounts of spatial and temporal redundancy, existing efficiency optimizations suffer from two major flaws:

  1. Coarse Granularity: Prior token-pruning or merging methods operate at a global, full-token level. This ignores fine-grained sub-token alignments caused by object motion or camera shifts, and it introduces high runtime overhead that disrupts hardware data locality.
  2. Static Metrics: Existing hardware accelerators rely on static importance metrics (like token magnitude) and completely ignore the cross-modal nature of VLMs—specifically, how the textual prompt dictates which visual regions actually matter.

Furthermore, these algorithmic approaches are rarely aligned with the hardware execution model of systolic arrays, which rely on small, regular General Matrix Multiplication (GEMM) tiles.

Methods

The authors introduce Focus, a Streaming Concentration Architecture that integrates seamlessly as a modular unit near the memory interface of a standard systolic array accelerator. It eliminates redundancy hierarchically using two main on-chip submodules:

  • Semantic Concentrator (SEC): This module performs token-level pruning within the attention layers. Instead of using static metrics, it leverages cross-modal attention scores to dynamically retain only the visual tokens relevant to the textual prompt. It utilizes a streaming importance analyzer and a lightweight, pipelined top-$k$ bubble sorter to prune uninformative tokens before memory write-back.
  • Similarity Concentrator (SIC): Operating within fully connected (FC) layers, the SIC targets fine-grained sub-token redundancy. It breaks token embeddings down into smaller vectors and uses a sliding window to perform localized similarity comparisons within spatiotemporal blocks.

To make these operations hardware-friendly, Focus introduces a convolution-style layout that maps tokens deterministically to unique memory banks. This design allows parallel similarity matching without memory bank conflicts or the need for expensive data replication.

Overview of the streaming multilevel concentration architecture

Datasets

The architecture was evaluated using three representative video-capable VLMs (LLaVA-OneVision-7B, LLaVA-Video-7B, and MiniCPM-V-2.6) across several prominent benchmarks:

  • Video Evaluation: VideoMME (VMME), MVBench (MVB), and MLVU.
  • Image Evaluation (for generalization testing): VQAv2, MME, and MMBench.

Key Insights and Ideas

  • Context-Driven Attention Shifts: Visual token importance is inherently tied to the prompt. For instance, given the same video frame, the model's attention shifts entirely depending on whether the question asks about a dog in the foreground or flowers in the background. Focus captures this dynamically.
  • Finer Granularity Unlocks More Sparsity: Full-token matching (e.g., 3584 dimensions) is often too restrictive to catch partial alignments. By dividing tokens into smaller 8-dimensional vectors, the authors discovered that over 64% of vectors exhibit a cosine similarity greater than 0.9, compared to only 18% at the full token level.
  • GEMM-Tile Friendly Compression: Rather than offloading compression to an external codec after writing full outputs to DRAM (which wastes bandwidth), Focus compresses data locally and on-chip immediately after each GEMM tile is produced.
Motivation for multilevel concentration showing prompt-aware attention maps and similarity distributions

Key Results

  • High Sparsity with Minimal Loss: Focus achieves an average computational sparsity of 80.19% (up to 83%), outperforming existing baselines like CMC and AdapTiV by over 30%. This compression incurs an average accuracy degradation of only 1.20%.
  • Hardware Efficiency: Compared to state-of-the-art accelerators, Focus delivers a 2.60× average speedup over AdapTiV and a 2.35× speedup over CMC. Against a vanilla systolic array, it achieves a 4.47× speedup and a 4.67× energy efficiency gain.
  • Reduced Memory Traffic: By sparsifying data on-chip before writing to memory, Focus compresses the input matrix by 5.6× and reduces off-chip DRAM traffic by 4.9×.
  • Minimal Hardware Overhead: The entire Focus unit is highly lightweight, consuming only 2.7% of the total systolic array area and increasing power consumption by a negligible 0.9%.

Limitations and Future Work

Currently, the value of $k$ in the top-$k$ semantic pruning configuration is determined by a predefined, layer-wise retention ratio optimized during a design space search. The authors note that future iterations could enhance this strategy by dynamically adapting the pruning threshold to the input context at runtime (e.g., using a post-softmax attention threshold or top-$p$ pruning), though this may introduce runtime variations across different inputs.

Summary updated

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance on tasks such as video captioning and visual question answering. However, their growing scale and video-level inputs lead to significant computational and memory overhead, posing challenges for real-time deployment on hardware accelerators. While prior work attempts to reduce redundancy via token pruning or merging, these methods typically operate at coarse granularity and incur high runtime overhead due to global token-level operations. In this study, we propose Focus, a Streaming Concentration Architecture that efficiently accelerates VLM inference through progressive, fine-grained redundancy elimination. Focus introduces a multilevel concentration paradigm that hierarchically compresses vision-language inputs at three levels: (1) semantic-guided token pruning based on textual prompts, (2) spatial-temporal block-level concentration using localized comparisons, and (3) vector-level redundancy removal via motion-aware matching. All concentration steps are tightly co-designed with the architecture to support streaming-friendly, on-chip execution. Focus leverages GEMM tiling, convolution-style layout, and cross-modal attention to minimize off-chip access while enabling high throughput. Implemented as a modular unit within a systolic-array accelerator, Focus achieves a 2.4x speedup and 3.3x reduction in energy, significantly outperforming state-of-the-art accelerators in both performance and energy efficiency. Full-stack implementation of Focus is open-sourced at https://github.com/dubcyfor3/Focus.
BibTeX
@misc{wei2025focus,
  author = {Chiyue Wei and Cong Guo and Junyao Zhang and Haoxuan Shan and Yifan Xu and Ziyue Zhang and Yudong Liu and Qinsi Wang and Changchun Zhou and Hai "Helen" Li and Yiran Chen},
  title = {{Focus: A Streaming Concentration Architecture for Efficient Vision-Language Models}},
  year = {2025},
  eprint = {2512.14661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.14661},
}
Figure