Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation

Bo Zhao, Quoc-Huy TrinharXiv 2025 · DOI · arXiv

TL;DR

This paper introduces Viper-F1, a novel multimodal large language model (MLLM) that significantly improves efficiency and fine-grained visual understanding by replacing traditional Transformer cross-attention with a Liquid State-Space Model (LSSM) and a novel Cross-Modal State-Space Modulator (CoSSM). This approach achieves competitive performance with much larger models while being significantly faster and more memory-efficient, making MLLMs more practical for resource-constrained environments.

What Problem Does This Paper Address?

Modern multimodal large language models (MLLMs) have shown remarkable capabilities in tasks like image captioning and visual question answering. However, their widespread adoption is hindered by high computational costs, primarily due to the quadratic complexity of the Transformer's cross-attention mechanism, which scales poorly with input sequence length. Furthermore, many existing efficient MLLMs struggle to precisely pinpoint task-relevant visual regions, leading to degraded performance on fine-grained reasoning tasks. This paper aims to bridge this gap by developing an MLLM that is both computationally efficient and capable of accurate, detailed visual grounding.

Methods

The core innovation of Viper-F1 lies in its novel architecture that replaces the computationally expensive Transformer cross-attention with a more efficient approach.

  • Liquid State-Space Model (LSSM): The paper integrates the Liquid Neural Network (LNN) [16], a type of state-space model (SSM), for sequence modeling. SSMs offer linear-time complexity, allowing them to efficiently process long sequences and capture long-range dependencies without the quadratic overhead of attention.
  • Cross-Modal State-Space Modulator (CoSSM): This is the key module for fusing visual and textual information. Instead of direct cross-attention, CoSSM uses a token-grid correlation mechanism to identify relevant visual patches for each text token. These correlations are then used to modulate the text representation via Feature-wise Linear Modulation (FiLM) [29]. The modulated text representations are then processed by an SSM to capture temporal and cross-modal dependencies. This approach allows for selective emphasis on visual regions relevant to the text prompt.
  • FiLM Conditioning: Feature-wise Linear Modulation is used twice: once to inject visual context into the text representation before the SSM, and again after the SSM to refine the output. This mechanism adaptively scales and shifts feature channels, allowing semantic cues from the image to influence the text representation without explicit token-level attention.
  • Vision Encoder: The SigLIP [32] model is used as the vision encoder to extract grid-level visual representations.
  • Language Decoder: The Liquid [35] model serves as the language decoder, processing the fused multimodal representation autoregressively.

Datasets

The paper uses the following datasets:

  • CC3M: Used for initializing the cross-modal fusion module during the first training stage.
  • LLaVA-CoT: Used for training the full model to enable reasoning abilities.
  • MMPR-v1.2: Processed into a chain-of-thought format for training.

The evaluation is performed on several standard benchmarks:

  • VQAv2
  • POPE
  • AI2D
  • MMMUval
  • MME
  • SQA-Image
  • MMB

Key Insights and Ideas

  • SSMs as a Viable Alternative to Attention: The paper strongly advocates for State-Space Models (SSMs) as a computationally efficient and effective alternative to Transformer attention for multimodal tasks, particularly for long sequences.
  • Fine-grained Grounding without Quadratic Cost: The CoSSM module's ability to precisely attend to task-relevant visual regions using token-grid correlations and FiLM modulation, while maintaining linear complexity, is a significant breakthrough for efficient MLLMs.
  • Efficiency Enables Practicality: By drastically reducing computational overhead, Viper-F1 makes advanced multimodal understanding accessible for resource-constrained devices and real-time applications.
  • Structured SSMs Outperform Basic SSMs: Ablation studies show that structured SSM variants (S4/S4D) are more effective than basic Mamba for capturing complex multimodal interactions.

Key Results

  • Performance: Viper-F1 achieves competitive or superior performance on benchmarks like VQAv2 and MME compared to much larger models (over 7B parameters), despite having fewer than 1B parameters. It also outperforms compact models in the 0.3B-3B range.
  • Efficiency: Viper-F1 demonstrates the lowest latency and highest throughput (46.67 tokens/sec) among lightweight multimodal baselines on the POPE dataset, highlighting its computational efficiency.
  • Qualitative Improvements: Qualitative comparisons show Viper-F1 generating more precise, instruction-aligned responses, accurately attending to fine-grained visual details that baselines miss.

Limitations and Future Work

  • Single-Image Input: The current model is designed for single-image inputs. Extending CoSSM to handle multiple images or video sequences is a significant area for future work.
  • Video Input: The authors acknowledge that they have not yet developed an effective version of CoSSM for video input. Future research will focus on developing efficient cross-modal connectors for temporal visual information.

Summary updated

BibTeX
@misc{zhao2025viper,
  author = {Bo Zhao and Quoc-Huy Trinh},
  title = {{Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation}},
  year = {2025},
  eprint = {2511.11177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.11177},
}
Figure