Towards Understanding Best Practices for Quantization of Vision-Language Models

Gautom Das, Vincent La, Ethan Lau, Abhinav Shrivastava, Matthew GwilliamarXiv 2026 · DOI

TL;DR

This paper investigates how to best quantize multimodal vision-language models (VLMs) to reduce their size and computational cost without sacrificing performance. They find that different components of these models have varying sensitivities to quantization, and that specialized quantization methods like GPTQ and AWQ are crucial for achieving high compression rates.

What Problem Does This Paper Address?

Large language models (LLMs) have revolutionized many AI tasks, but their massive size and computational demands make them expensive to train and deploy. Multimodal LLMs (MLLMs), which combine LLMs with vision encoders, exacerbate these issues. Quantization, a technique to reduce model precision, is a promising solution for making these models more accessible and efficient. However, simply applying existing quantization methods to MLLMs doesn't guarantee optimal results. This paper aims to understand the best practices for quantizing MLLMs by exploring how different quantization strategies affect various components of these complex architectures and how these effects vary across different tasks.

Methods

The researchers employ a multi-pronged approach:

  • Uniform Quantization: They first explore a basic k-bit uniform quantization strategy to build intuition. This involves systematically quantizing different parts of the MLLM (components like the vision encoder or LLM, block groups within these components, and specific layer types like attention or feed-forward networks) across a range of bit widths.
  • State-of-the-Art Quantization: They then apply advanced post-training quantization (PTQ) methods, specifically GPTQ and AWQ, which are designed for LLMs. These methods are known to be more effective than uniform quantization at preserving performance at lower bit widths.
  • Component Importance Analysis: To understand why certain quantization strategies work better than others, they develop a framework to quantify the importance of each model component (ViT, Q-Former, LLM) under quantization. This framework uses three complementary tree-based importance analysis techniques: Random Forest Feature Importance, Permutation Feature Importance, and SHAP values, combined into a consensus ranking.

Datasets

The paper evaluates their methods on several standard datasets:

  • COCO Captions: For image captioning tasks.
  • VQAv2: For visual question answering.
  • GQA: Another visual question answering dataset, noted for being a successor to prior VQA datasets that had biases.
  • Flickr (Text-to-Image Retrieval): For retrieval tasks.

Key Insights and Ideas

  • Component Sensitivity Varies: Not all parts of an MLLM are equally sensitive to quantization. The LLM component is often the most critical, especially for tasks requiring complex reasoning.
  • SOTA Quantization is Key: Advanced methods like GPTQ and AWQ significantly outperform uniform quantization in preserving performance at lower bit widths (e.g., 3.5-4.5 bits per weight).
  • Task Dictates Strategy: The nature of the task (e.g., captioning vs. question answering vs. retrieval) strongly influences which components are most important to preserve during quantization. Reasoning-heavy tasks like VQA benefit most from preserving LLM precision.
  • Quantization Method Matters: The choice of quantization method (GPTQ vs. AWQ) can dramatically shift component importance. AWQ tends to focus quantization efforts on the LLM, while GPTQ distributes importance more evenly.
  • Architecture Interplay: The way components are connected and their roles within the architecture (e.g., a Q-Former's position in retrieval tasks or a projection layer's role in LLaVA) also influence quantization patterns and component importance.

Key Results

  • High Compression Achieved: SOTA quantization methods (GPTQ and AWQ) enable performance-preserving quantization down to 3.5-4.5 bits per weight (bpw) for MLLMs, a significant improvement over uniform quantization which typically requires 6.0-10.0 bpw for comparable performance.
  • LLM Dominance in VQA: For VQA tasks, the LLM component is overwhelmingly the most important, with AWQ achieving over 90% importance for the LLM in LLaVA.
  • Balanced Importance in Captioning: For COCO captioning, GPTQ shows a more balanced importance distribution between the ViT and LLM (around 50% each), while AWQ still heavily favors the LLM.
  • ViT Importance in Retrieval: In retrieval tasks (which don't use the LLM), the ViT component becomes dominant, with GPTQ showing over 70% importance.
  • Interactions Worsen Performance: Quantizing multiple components simultaneously can sometimes lead to worse performance than quantizing individual components, highlighting the need for careful, holistic optimization.

Limitations and Future Work

  • Simulated Quantization: The study focuses on simulated quantization and does not directly measure end-to-end latency or hardware-specific optimizations. Future work could explore these aspects.
  • Limited Architectures: While BLIP-2 and LLaVA are representative, future work could investigate a broader range of MLLM architectures.
  • Hardware-Specific Optimizations: The paper suggests that hardware-specific optimizations could further improve deployment efficiency.
  • Broader Modalities: The methodology developed for analyzing component importance could be extended to MLLMs incorporating other modalities beyond vision and language.

Summary updated

Abstract

Large language models (LLMs) deliver impressive results for a variety of tasks, but state-of-the-art systems require fast GPUs with large amounts of memory. To reduce both the memory and latency of these systems, practitioners quantize their learned parameters, typically at half precision. A growing body of research focuses on preserving the model performance with more aggressive bit widths, and some work has been done to apply these strategies to other models, like vision transformers. In our study we investigate how a variety of quantization methods, including state-of-the-art GPTQ and AWQ, can be applied effectively to multimodal pipelines comprised of vision models, language models, and their connectors. We address how performance on captioning, retrieval, and question answering can be affected by bit width, quantization method, and which portion of the pipeline the quantization is used for. Results reveal that ViT and LLM exhibit comparable importance in model performance, despite significant differences in parameter size, and that lower-bit quantization of the LLM achieves high accuracy at reduced bits per weight (bpw). These findings provide practical insights for efficient deployment of MLLMs and highlight the value of exploration for understanding component sensitivities in multimodal models. Our code is available at https://github.com/gautomdas/mmq.
BibTeX
@misc{das2026towards,
  author = {Gautom Das and Vincent La and Ethan Lau and Abhinav Shrivastava and Matthew Gwilliam},
  title = {{Towards Understanding Best Practices for Quantization of Vision-Language Models}},
  year = {2026},
  eprint = {2601.15287},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.15287},
}
Figure