TL;DR
MoMoVL is a novel mixed-precision quantization framework designed to bring large Mixture-of-Experts (MoE) Vision-Language Models (VLMs) to resource-constrained edge devices. By recognizing that experts in these models specialize by modality—and that text-centric experts are significantly more sensitive to quantization noise than visual-centric ones—MoMoVL intelligently allocates bit-widths to maximize accuracy within a strict memory budget.
What Problem Does This Paper Address?
The "agentic web"—where AI agents autonomously navigate and interact with web content—requires powerful Vision-Language Models (VLMs) to run locally on devices for privacy and low latency. Modern VLMs often use a Mixture-of-Experts (MoE) architecture. While MoE is computationally efficient because it only activates a fraction of its parameters for any given input, it is a memory hog: every single expert must reside in RAM, regardless of whether it's active.
To fit these models onto phones or laptops, researchers use quantization (reducing the precision of weights, e.g., from 16-bit to 4-bit or 2-bit). However, standard quantization methods treat all experts as equally important or only look at how often they are used. The authors of MoMoVL argue this is suboptimal for multimodal models because it ignores modality-driven specialization. Visual tokens are often spatially redundant, while text tokens carry high information density. Consequently, losing precision in an expert that handles text hurts the model much more than losing precision in one that handles images.
Methods
MoMoVL introduces a modality-aware, adaptive compression workflow consisting of three main stages:
- Modality-Aware Activation Profiling: The model is run on a small multimodal calibration set. The framework records which experts are triggered by visual tokens versus text tokens, creating a "specialization map" for the model.
- Measuring Modality-Specific Sensitivity: To quantify how much quantization noise affects each modality, the authors use the gradients of the Supervised Fine-Tuning (SFT) loss. They found that text tokens consistently exhibit much higher gradients (roughly 5.5x higher on average) than visual tokens, indicating they are more "sensitive" to perturbations.
- Adaptive Bit-Width Assignment: MoMoVL combines the activation profile and the sensitivity scale into a single importance metric per expert. It then uses a greedy algorithm to assign higher bit-widths (e.g., INT4) to the most "important" experts and lower bit-widths (e.g., INT2) to the rest, staying within a user-defined memory budget.
The architectural context of how these tokens are routed to experts is shown in Figure 1.
Datasets
- Calibration: 128 image-caption pairs randomly sampled from the ShareGPT4V dataset (built on COCO).
- Evaluation: The framework was tested across eight diverse multimodal benchmarks: MMMU, MMBench-V1.1, SEEDBench, ScienceQA, RealWorldQA, MME-RealWorld, AI2D, and OCRBench.
- Models: Evaluation focused on two popular MoE-based VLMs: MolmoE-1B (7.2B total parameters) and DeepSeek-VL2-Tiny (3.4B total parameters).
Key Insights and Ideas
- Experts are Specialists: The paper empirically proves that experts in VLMs are not generalists. As seen in Figure 2, there is a clear divide between visual-centric and text-centric experts.
- Text is the "Semantic Anchor": Even though visual tokens vastly outnumber text tokens in a typical prompt, the text tokens are far more critical for maintaining model accuracy. Prioritizing text-centric experts for higher precision improved accuracy by up to 4.98% compared to prioritizing visual-centric ones at the same compression level.
- The Vanishing Visual Gradient: At the very last layer of the model, the sensitivity to visual tokens drops to zero. This is because the model is focused entirely on generating the next text token, meaning the final layer's experts should be quantized based almost exclusively on their text-processing importance.
Key Results
- Memory Efficiency: MoMoVL reduced the memory footprint of MolmoE-1B by 15.44% compared to a uniform INT4 baseline, with only a modest drop in accuracy.
- Superior Trade-offs: It consistently outperformed state-of-the-art mixed-precision baselines (like MoE-APEX and Fate) that do not account for modality.
- Sensitivity Convergence: The authors found that expert importance metrics converge very quickly, meaning the framework only needs a tiny amount of calibration data (as few as 128 samples) to be effective.
Limitations and Future Work
- Model Scale: Due to hardware constraints, the study focused on models under 10B parameters. While the authors expect the findings to generalize to larger models, this remains to be verified.
- Quantization Granularity: The current work primarily explores switching between INT2 and INT4. Future work could incorporate a broader range of bit-widths (e.g., 3-bit or 1-bit) for even finer control.
- Static vs. Dynamic: MoMoVL currently generates a static quantization profile. A dynamic version that adjusts bit-widths at runtime based on specific input content could potentially push the efficiency envelope even further.
Summary updated