TL;DR
Focus is a hardware-software co-designed architecture that accelerates Vision-Language Model (VLM) inference by progressively eliminating visual redundancy at three distinct levels: semantic, block, and vector. Operating entirely on-chip in a streaming fashion, Focus achieves a 2.4× speedup and a 3.3× reduction in energy consumption over state-of-the-art accelerators while maintaining high model accuracy.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) excel at multimodal tasks like video captioning and visual question answering, but their deployment on edge devices is severely bottlenecked by computational and memory overhead. In video-based VLMs, visual tokens make up a staggering 98% to 99% of the total input sequence. While video frames contain massive amounts of spatial and temporal redundancy, existing efficiency optimizations suffer from two major flaws:
- Coarse Granularity: Prior token-pruning or merging methods operate at a global, full-token level. This ignores fine-grained sub-token alignments caused by object motion or camera shifts, and it introduces high runtime overhead that disrupts hardware data locality.
- Static Metrics: Existing hardware accelerators rely on static importance metrics (like token magnitude) and completely ignore the cross-modal nature of VLMs—specifically, how the textual prompt dictates which visual regions actually matter.
Furthermore, these algorithmic approaches are rarely aligned with the hardware execution model of systolic arrays, which rely on small, regular General Matrix Multiplication (GEMM) tiles.
Methods
The authors introduce Focus, a Streaming Concentration Architecture that integrates seamlessly as a modular unit near the memory interface of a standard systolic array accelerator. It eliminates redundancy hierarchically using two main on-chip submodules:
- Semantic Concentrator (SEC): This module performs token-level pruning within the attention layers. Instead of using static metrics, it leverages cross-modal attention scores to dynamically retain only the visual tokens relevant to the textual prompt. It utilizes a streaming importance analyzer and a lightweight, pipelined top-$k$ bubble sorter to prune uninformative tokens before memory write-back.
- Similarity Concentrator (SIC): Operating within fully connected (FC) layers, the SIC targets fine-grained sub-token redundancy. It breaks token embeddings down into smaller vectors and uses a sliding window to perform localized similarity comparisons within spatiotemporal blocks.
To make these operations hardware-friendly, Focus introduces a convolution-style layout that maps tokens deterministically to unique memory banks. This design allows parallel similarity matching without memory bank conflicts or the need for expensive data replication.
Datasets
The architecture was evaluated using three representative video-capable VLMs (LLaVA-OneVision-7B, LLaVA-Video-7B, and MiniCPM-V-2.6) across several prominent benchmarks:
- Video Evaluation: VideoMME (VMME), MVBench (MVB), and MLVU.
- Image Evaluation (for generalization testing): VQAv2, MME, and MMBench.
Key Insights and Ideas
- Context-Driven Attention Shifts: Visual token importance is inherently tied to the prompt. For instance, given the same video frame, the model's attention shifts entirely depending on whether the question asks about a dog in the foreground or flowers in the background. Focus captures this dynamically.
- Finer Granularity Unlocks More Sparsity: Full-token matching (e.g., 3584 dimensions) is often too restrictive to catch partial alignments. By dividing tokens into smaller 8-dimensional vectors, the authors discovered that over 64% of vectors exhibit a cosine similarity greater than 0.9, compared to only 18% at the full token level.
- GEMM-Tile Friendly Compression: Rather than offloading compression to an external codec after writing full outputs to DRAM (which wastes bandwidth), Focus compresses data locally and on-chip immediately after each GEMM tile is produced.
Key Results
- High Sparsity with Minimal Loss: Focus achieves an average computational sparsity of 80.19% (up to 83%), outperforming existing baselines like CMC and AdapTiV by over 30%. This compression incurs an average accuracy degradation of only 1.20%.
- Hardware Efficiency: Compared to state-of-the-art accelerators, Focus delivers a 2.60× average speedup over AdapTiV and a 2.35× speedup over CMC. Against a vanilla systolic array, it achieves a 4.47× speedup and a 4.67× energy efficiency gain.
- Reduced Memory Traffic: By sparsifying data on-chip before writing to memory, Focus compresses the input matrix by 5.6× and reduces off-chip DRAM traffic by 4.9×.
- Minimal Hardware Overhead: The entire Focus unit is highly lightweight, consuming only 2.7% of the total systolic array area and increasing power consumption by a negligible 0.9%.
Limitations and Future Work
Currently, the value of $k$ in the top-$k$ semantic pruning configuration is determined by a predefined, layer-wise retention ratio optimized during a design space search. The authors note that future iterations could enhance this strategy by dynamically adapting the pruning threshold to the input context at runtime (e.g., using a post-softmax attention threshold or top-$p$ pruning), though this may introduce runtime variations across different inputs.
Summary updated