MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models

Dongkeun Lee, Wonjun LeearXiv 2026 · DOI · arXiv

TL;DR

MoMoVL is a novel mixed-precision quantization framework designed to bring large Mixture-of-Experts (MoE) Vision-Language Models (VLMs) to resource-constrained edge devices. By recognizing that experts in these models specialize by modality—and that text-centric experts are significantly more sensitive to quantization noise than visual-centric ones—MoMoVL intelligently allocates bit-widths to maximize accuracy within a strict memory budget.

What Problem Does This Paper Address?

The "agentic web"—where AI agents autonomously navigate and interact with web content—requires powerful Vision-Language Models (VLMs) to run locally on devices for privacy and low latency. Modern VLMs often use a Mixture-of-Experts (MoE) architecture. While MoE is computationally efficient because it only activates a fraction of its parameters for any given input, it is a memory hog: every single expert must reside in RAM, regardless of whether it's active.

To fit these models onto phones or laptops, researchers use quantization (reducing the precision of weights, e.g., from 16-bit to 4-bit or 2-bit). However, standard quantization methods treat all experts as equally important or only look at how often they are used. The authors of MoMoVL argue this is suboptimal for multimodal models because it ignores modality-driven specialization. Visual tokens are often spatially redundant, while text tokens carry high information density. Consequently, losing precision in an expert that handles text hurts the model much more than losing precision in one that handles images.

Methods

MoMoVL introduces a modality-aware, adaptive compression workflow consisting of three main stages:

  1. Modality-Aware Activation Profiling: The model is run on a small multimodal calibration set. The framework records which experts are triggered by visual tokens versus text tokens, creating a "specialization map" for the model.
  2. Measuring Modality-Specific Sensitivity: To quantify how much quantization noise affects each modality, the authors use the gradients of the Supervised Fine-Tuning (SFT) loss. They found that text tokens consistently exhibit much higher gradients (roughly 5.5x higher on average) than visual tokens, indicating they are more "sensitive" to perturbations.
  3. Adaptive Bit-Width Assignment: MoMoVL combines the activation profile and the sensitivity scale into a single importance metric per expert. It then uses a greedy algorithm to assign higher bit-widths (e.g., INT4) to the most "important" experts and lower bit-widths (e.g., INT2) to the rest, staying within a user-defined memory budget.

The architectural context of how these tokens are routed to experts is shown in Figure 1.

Figure 1: Architectural overview of an MoE-based VLM. Visual tokens and text tokens are processed with shared and routed experts using top-k routing.

Datasets

  • Calibration: 128 image-caption pairs randomly sampled from the ShareGPT4V dataset (built on COCO).
  • Evaluation: The framework was tested across eight diverse multimodal benchmarks: MMMU, MMBench-V1.1, SEEDBench, ScienceQA, RealWorldQA, MME-RealWorld, AI2D, and OCRBench.
  • Models: Evaluation focused on two popular MoE-based VLMs: MolmoE-1B (7.2B total parameters) and DeepSeek-VL2-Tiny (3.4B total parameters).

Key Insights and Ideas

  • Experts are Specialists: The paper empirically proves that experts in VLMs are not generalists. As seen in Figure 2, there is a clear divide between visual-centric and text-centric experts.
  • Text is the "Semantic Anchor": Even though visual tokens vastly outnumber text tokens in a typical prompt, the text tokens are far more critical for maintaining model accuracy. Prioritizing text-centric experts for higher precision improved accuracy by up to 4.98% compared to prioritizing visual-centric ones at the same compression level.
  • The Vanishing Visual Gradient: At the very last layer of the model, the sensitivity to visual tokens drops to zero. This is because the model is focused entirely on generating the next text token, meaning the final layer's experts should be quantized based almost exclusively on their text-processing importance.
Figure 2: Modality-specific expert activation in MolmoE-1B on the COCO caption dataset. (a-b) Each cell shows the percentage of visual/text tokens routed to a given expert (summing to 800 % per layer due to top-8 routing); darker colors indicate more frequent activations. (c) Difference in percentage points; red and blue mark visual- and text-centric experts, respectively. All three panels share one color scale (0-61.5 %, ±61.5 %p in (c)); the first 16 of 64 experts in native index order are shown.

Key Results

  • Memory Efficiency: MoMoVL reduced the memory footprint of MolmoE-1B by 15.44% compared to a uniform INT4 baseline, with only a modest drop in accuracy.
  • Superior Trade-offs: It consistently outperformed state-of-the-art mixed-precision baselines (like MoE-APEX and Fate) that do not account for modality.
  • Sensitivity Convergence: The authors found that expert importance metrics converge very quickly, meaning the framework only needs a tiny amount of calibration data (as few as 128 samples) to be effective.

Limitations and Future Work

  • Model Scale: Due to hardware constraints, the study focused on models under 10B parameters. While the authors expect the findings to generalize to larger models, this remains to be verified.
  • Quantization Granularity: The current work primarily explores switching between INT2 and INT4. Future work could incorporate a broader range of bit-widths (e.g., 3-bit or 1-bit) for even finer control.
  • Static vs. Dynamic: MoMoVL currently generates a static quantization profile. A dynamic version that adjusts bit-widths at runtime based on specific input content could potentially push the efficiency envelope even further.

Summary updated

Abstract

The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models are widely adopted in modern VLMs for compute-efficient scalability, yet their multi-expert architecture poses deployment challenges due to substantial memory demands. Mixed-precision quantization is an effective compression technique that reduces bit-widths while adapting to the heterogeneous hardware budgets of edge devices, particularly well-suited for MoE due to its inherent structural sparsity, which causes imbalanced importance among experts. While prior work has applied quantization for efficient MoE inference at the edge, we point out that in MoE-based VLMs, the modality-specific characteristics of visual and text tokens further impact expert quantization. To address this issue, we present MoMoVL, a modality-aware adaptive compression framework for MoE-based VLMs that brings vision-language inference within reach of on-device resource budgets. Motivated by our empirical findings on modality-induced disparities across experts, MoMoVL jointly exploits the per-expert activation profile and per-modality quantization sensitivity to guide bit-width assignment. Experimental results show that MoMoVL improves the accuracy-compression trade-off relative to state-of-the-art mixed-precision baselines and reduces memory footprint by up to 15.44% with a modest drop in accuracy over its uniform-precision baseline. Our code and data are publicly available at https://github.com/sco-edge/MoMoVL.
BibTeX
@misc{lee2026momovl,
  author = {Dongkeun Lee and Wonjun Lee},
  title = {{MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models}},
  year = {2026},
  eprint = {2611.23695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23695},
}

Loading related content...

Figure