High Efficiency Image Compression for Large Visual-Language Models

Binzhe Li, Shurun Wang, Shiqi Wang, Yan Ye

This paper, recently submitted to IEEE Transactions on Circuits and Systems for Video Technology, tackles a fundamental shift in how we think about data: what happens when the "viewer" of an image isn't a human, but a Large Visual-Language Model (LVLM)?

TL;DR

Researchers have developed a new image compression framework specifically optimized for Large Visual-Language Models (LVLMs) rather than human eyes. By using "semantic tokens" from large models to guide the compression process, they achieved over 50% bitrate savings compared to the state-of-the-art VVC standard while maintaining high accuracy across tasks like image captioning and visual grounding.

What Problem Does This Paper Address?

For decades, image compression standards like JPEG and VVC (Versatile Video Coding) have been designed for the Human Visual System (HVS). They work by discarding information that the human eye can't easily perceive.

However, we are entering an era where AI models—specifically LVLMs like GPT-4V or open-source equivalents—are the primary consumers of visual data. These models "see" differently than we do. A texture that looks "noisy" to a human might contain vital semantic cues for an AI, while a high-resolution background that looks "beautiful" to us might be useless "bitrate-sink" for a machine task.

Previous "Compression for Machines" (VCM) research often focused on narrow tasks like object detection. This paper addresses the need for a generalized compression framework that works for the broad, multi-modal capabilities of modern LVLMs.

Methods

The authors propose a two-stage framework:

  1. Pre-editing Module: This isn't just a simple filter. It’s a neural network that uses "semantic tokens" (internal representations from a Vision Transformer) as prompts. It effectively "re-paints" the image to highlight features the LVLM cares about and suppresses irrelevant data before it even hits the compressor.
  2. Variable Bitrate End-to-End Codec: A deep-learning-based encoder/decoder pair that can adjust its compression strength (bitrate) dynamically.

The Secret Sauce: Token-Level Optimization Instead of just measuring pixel-to-pixel "distortion" (like MSE or PSNR), the authors introduced a novel loss function based on:

  • Token Distortion: Ensuring the internal features the model extracts from the compressed image match the original.
  • Rank Loss: A clever mathematical approach using the "rank" of semantic tokens. The idea is that the rank reflects "semantic abundance"—by preserving the rank of these tokens during compression, the framework ensures the model doesn't lose the "richness" of the scene's information.

Datasets

The framework was trained and evaluated using several industry-standard benchmarks:

  • COCO (Common Objects in Context): Used for general training and image captioning/retrieval tasks.
  • RefCOCO / RefCOCO+: Used for "Visual Grounding" (the ability of a model to locate a specific object based on a text description).
  • Task Models: The researchers tested their results against high-performing LVLMs like OFA and One-Peace (OP) to ensure generalization.

Key Insights and Ideas

  • Machines Don't Need "Pretty" Pixels: The qualitative results show that the "pre-edited" images sometimes look slightly different to humans (e.g., sharper boundaries or simplified textures), but these changes are exactly what help the LVLM focus on the "what" and "where" of the scene.
  • Semantic Tokens as a Universal Language: By optimizing for tokens rather than specific task labels (like "dog" or "cat"), the compression becomes "task-agnostic." It works just as well for captioning as it does for complex visual reasoning.
  • The Power of Rank: The use of rank-based loss is a significant theoretical contribution. It treats semantic information as a structural property of the data rather than just a collection of values, making the compression more robust to noise.

Key Results

  • Massive Efficiency Gains: The proposed method achieved BD-Rate savings of 50% to 63% compared to the VVC anchor. In plain English: it can provide the same level of AI accuracy using less than half the data.
  • Superior Accuracy at Low Bitrates: At very low bitrates where VVC-compressed images become "blurry" and cause AI accuracy to plummet, the proposed method maintains high performance.
  • Speed: The decoder inference time is approximately 20ms on an NVIDIA A100, suggesting it is fast enough for real-world cloud-based AI pipelines.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Complexity: The system requires running a "token extractor" (a large model itself) during the training phase, which is computationally expensive, though this is a one-time cost for training.
  • Non-Monotonic Curves: In some specific tests, the Rate-Accuracy (RA) curves weren't perfectly smooth, suggesting that the relationship between bitrate and AI "understanding" is more complex than traditional human-centric metrics.
  • Future Expansion: The authors aim to expand this to video data and even more diverse multi-modal tasks, potentially moving toward a world where the entire internet's visual backbone is optimized for the AI models that process it.

Summary updated

Abstract

In recent years, large visual language models (LVLMs) have shown impressive performance and promising generalization capability in multi-modal tasks, thus replacing humans as receivers of visual information in various application scenarios. In this paper, we pioneer to propose a variable bitrate image compression scheme consisting of a pre-editing module and an end-to-end codec to achieve promising rate-accuracy performance for different LVLMs. In particular, instead of optimizing an adaptive pre-editing network towards a particular task or several representative tasks, we propose a new optimization strategy tailored for LVLMs, which is designed based on the representation and discrimination capability with token-level distortion and rank. The pre-editing module and the variable bitrate end-to-end image codec are jointly trained by the losses based on semantic tokens of the large model, which introduce enhanced generalization capability for various data and tasks. Experimental results demonstrate that the proposed framework could efficiently achieve much better rate-accuracy performance compared to the state-of-the-art coding standard, Versatile Video Coding. Meanwhile, experiments with multi-modal tasks have revealed the robustness and generalization capability of the proposed framework.
BibTeX
@article{li2025high,
  author = {Binzhe Li and Shurun Wang and Shiqi Wang and Yan Ye},
  title = {{High Efficiency Image Compression for Large Visual-Language Models}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  doi = {10.1109/tcsvt.2024.3488181},
}

Loading related content...

Figure