VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression

Kyle Sargent, Ruiqi Gao, Philipp Henzler, Charles Herrmann, Aleksander Holynski, Li Fei-Fei, Jiajun Wu, Jason Y. Zhang

Introduction

Image compression has traditionally relied on mathematical metrics like Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR). While these metrics are easy to compute, they often fail to capture human visual preferences. A sharp image with minor color shifts might be penalized more than a blurry one, despite being preferred by human viewers. VLIC (Vision-Language Models for Image Compression) addresses this "perception gap" by leveraging the zero-shot reasoning capabilities of Vision-Language Models (VLMs) to act as perceptual judges during the training process.

The Perception Gap in Compression

Standard perceptual losses, such as LPIPS, use neural networks trained on human judgments to bridge the gap. However, these metrics can be "gamed" by generative models, leading to unnatural artifacts. More importantly, they lack semantic awareness; they cannot distinguish between a critical distortion in a human face and a minor change in a background texture. VLIC introduces a way to distill high-level visual common sense into compression models.

Method: VLM-Guided Alignment

The VLIC framework employs a two-stage training pipeline for a diffusion-based autoencoder based on the FlowMo architecture. After an initial pre-training phase using standard MSE and LPIPS objectives, the model undergoes a post-training phase using Diffusion Direct Preference Optimization (DPO).

The VLIC training pipeline using Diffusion DPO and VLM feedback.

In this stage, the system generates two candidate reconstructions (A and B) for a given compressed code. A VLM (specifically Gemini 2.5-Flash) evaluates these against the original image. The VLM is prompted to reason about specific details—such as the clarity of a license plate or the sharpness of foliage—and provide a preference. To ensure a robust training signal, the authors implement "Self-Ensembling" and a "Unanimous Vote" strategy, where a preference is only accepted if the VLM's choice aligns with traditional perceptual metrics like LPIPS. This prevents the model from over-optimizing for a single judge.

Experimental Setup and Datasets

The model was evaluated on several benchmark datasets, including MS-COCO, CLIC 2020, and CLIC 2022. To validate the VLM's effectiveness as a judge, the researchers also collected a new dataset of 5,401 human preference judgments. They found that Gemini 2.5-Flash could replicate human binary choices with accuracy comparable to specialized perceptual metrics, with the added benefit of providing semantic reasoning for its decisions.

Key Results

VLIC achieves state-of-the-art performance across multiple human-centric evaluation metrics. In large-scale user studies, VLIC significantly outperformed established baselines like HiFiC and PerCo in terms of Human Elo scores. On MS-COCO, VLIC achieved an Elo score of 1112, compared to 1103 for LPIPS-only training.

Qualitative comparison showing improved text and detail reconstruction with DPO.

Qualitative analysis reveals that VLIC is particularly effective at preserving human-salient features. As shown in the comparison, the DPO-aligned model produces much clearer text on street signs and more realistic textures in complex scenes compared to models trained without VLM guidance. Furthermore, VLIC showed substantial improvements in distributional metrics like FID and FD-DINO, indicating a higher degree of overall realism.

Conclusion and Future Work

VLIC demonstrates that VLMs can serve as powerful "perceptual teachers" for generative tasks. By using DPO to align diffusion models with VLM preferences, the researchers have created a compression system that prioritizes the visual details humans care about most. While challenges remain regarding the computational cost of diffusion models and VLM inference, this work opens the door for more semantically aware and human-aligned media processing. Future research may explore even more specialized alignment, such as prioritizing legibility in document compression or facial fidelity in video calls.

Summary updated

BibTeX
@inproceedings{sargent2026vlic,
  author = {Kyle Sargent and Ruiqi Gao and Philipp Henzler and Charles Herrmann and Aleksander Holynski and Li Fei-Fei and Jiajun Wu and Jason Y. Zhang},
  title = {{VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=Xwzq032zWs},
}
Figure