Abstract
In recent years, large visual language models (LVLMs) have shown impressive performance and promising generalization capability in multi-modal tasks, thus replacing humans as receivers of visual information in various application scenarios. In this paper, we pioneer to propose a variable bitrate image compression framework consisting of a pre-editing module and an end-to-end codec to achieve promising rate-accuracy performance for different LVLMs. In particular, instead of optimizing an adaptive pre-editing network towards a particular task or several representative tasks, we propose a new optimization strategy tailored for LVLMs, which is designed based on the representation and discrimination capability with token-level distortion and rank. The pre-editing module and the variable bitrate end-to-end image codec are jointly trained by the losses based on semantic tokens of the large model, which introduce enhanced generalization capability for various data and tasks. Experimental results demonstrate that the proposed framework could efficiently achieve much better rate-accuracy performance compared to the state-of-the-art coding standard, Versatile Video Coding. Meanwhile, experiments with multi-modal tasks have revealed the robustness and generalization capability of the proposed framework.
Introduction
Traditional image compression standards like JPEG and VVC (Versatile Video Coding) are fundamentally designed for the Human Visual System (HVS). They prioritize discarding information imperceptible to the human eye. However, as Large Visual-Language Models (LVLMs) like GPT-4V and open-source counterparts become primary consumers of visual data, this paradigm must shift. Machines "see" differently; textures perceived as noise by humans might hold critical semantic cues for an AI, while high-resolution backgrounds might be redundant for machine tasks. This paper addresses the need for a generalized compression framework optimized for the broad, multi-modal capabilities of modern LVLMs.
Proposed Framework
The proposed architecture consists of two primary components: a Token-based Pre-editing Module and a Variable Bitrate End-to-End Codec.
The pre-editing module is a neural network that utilizes "semantic tokens"—internal representations from a Vision Transformer—as prompts. It effectively "re-paints" the image to emphasize features critical to the LVLM while suppressing irrelevant data before compression. The variable bitrate codec then dynamically adjusts compression strength to meet specific bitrate targets.
Optimization Strategy: Token-Level Guidance
A key innovation of this work is the optimization strategy. Rather than relying on pixel-level metrics like Mean Squared Error (MSE), the framework is trained using:
- Token Distortion Loss: Ensures that internal features extracted from the compressed image align with those of the original.
- Rank Loss: Preserves the "rank" of semantic tokens, which reflects semantic abundance. By maintaining this structural property, the framework ensures the model retains the richness of the scene's information.
This approach makes the compression "task-agnostic," allowing it to perform well across diverse applications such as image captioning, visual grounding, and text-image retrieval.
Experimental Results
The framework was evaluated on benchmarks including COCO and RefCOCO/RefCOCO+, testing against high-performing LVLMs like OFA and One-Peace (OP).
Key findings include:
- Efficiency: The method achieves BD-Rate savings of 50% to 63% compared to the VVC anchor, providing equivalent AI accuracy at half the data rate.
- Robustness at Low Bitrates: While VVC performance drops sharply at low bitrates due to blurring, the proposed method maintains high accuracy.
- Speed: With a decoder inference time of approximately 20ms on an NVIDIA A100, the system is viable for real-world cloud-based AI pipelines.
Conclusion
This research demonstrates that optimizing compression for machine "understanding" rather than human "perception" yields significant efficiency gains. By using semantic tokens as a universal language for optimization, the framework achieves state-of-the-art performance across various multi-modal tasks. Future work will explore extending these principles to video data and a wider array of complex visual reasoning tasks.
Summary updated