LG-VQ: Language-Guided Codebook Learning

Guotao Liang, Baoquan Zhang, Yaowei Wang, Xutao Li, Yunming Ye, Huaibin Wang, Chuyao Luo, Kola Ye, linfeng Luo

TL;DR

LG-VQ (Language-Guided Vector Quantization) is a novel framework that bridges the "modal gap" in image synthesis by aligning discrete visual codebooks with language semantics. By leveraging pre-trained text encoders and two new alignment modules, it enables VQ models to learn codebooks that are not just good at reconstructing pixels, but also rich in high-level semantic knowledge.

What Problem Does This Paper Address?

Vector Quantization (VQ) is the backbone of modern high-fidelity image synthesis (like VQ-VAE and VQ-GAN). It works by compressing images into a sequence of discrete "tokens" from a learned codebook. However, traditional VQ methods are single-modal: they focus almost entirely on low-level visual features like edges, textures, and colors.

When these codebooks are used for multi-modal tasks—such as generating an image from a text prompt or answering a question about a photo—they often struggle. This is because there is a significant "modal gap" between the visual tokens and the language used to describe them. A standard codebook might know how to draw a "racket," but it doesn't inherently understand the semantic relationship between a "woman" and a "racket" in the context of "holding."

Methods

The authors propose LG-VQ, a model-agnostic framework that can be integrated into existing VQ architectures to inject language priors into the codebook learning process.

Figure 2: The overall architecture of the proposed LG-VQ method. The right part of the figure is the basic VQ-VAE module, left is our language-guided module, which consists of three losses: global semantic alignment (Lgsa), masked text prediction (Lmtp), and relationship alignment supervision (Lras). Here, pre-trained text information guides discrete code tokens of an image to learning rich semantic knowledge based on three losses.

The framework introduces three specific training objectives:

  1. Semantic Alignment Module (SAM):
    • Global Semantic Alignment ($L_{gsa}$): Uses a contrastive loss (InfoNCE) to ensure the global representation of an image's visual codes matches the global embedding of its corresponding text description, using a pre-trained CLIP model as a teacher.
  2. Masked Text Prediction ($L_{mtp}$): In a "fill-in-the-blanks" task, the model must use visual codes to predict words that have been masked out of the image's caption. This forces the codebook to capture fine-grained semantic details.
  3. Relationship Alignment Module (RAM):
    • Relationship Alignment Supervision ($L_{ras}$): This is the most novel component. It calculates the semantic similarity between pairs of words in a sentence (e.g., "girl" and "earring") and enforces that the visual codes corresponding to those words maintain a similar relationship in the visual feature space.
Figure 3: Illustration of relationship align- ment module, we use Zvt to align with two words, then inject the semantic relationship of two words into Z codes.

Datasets

The model was evaluated across four diverse public datasets to test its generality:

  • TextCaps: Focuses on images containing text, requiring complex reading and reasoning.
  • CelebA-HQ: High-quality human face images.
  • CUB-200: Fine-grained bird species images and descriptions.
  • MS-COCO: A standard large-scale benchmark for general object detection and captioning.

Key Insights and Ideas

  • Beyond Pixels: The core "aha moment" is that a codebook shouldn't just be a compression tool; it should be a semantic bridge. By forcing visual tokens to "explain" text, they become much more useful for downstream reasoning.
  • Relationships are Key: Simply aligning "cat" with a cat-token isn't enough. The authors show that capturing the interaction between entities (the "Relationship Alignment") is crucial for complex tasks like Visual Question Answering (VQA).
  • Model-Agnosticism: LG-VQ isn't a new VQ model; it's a training strategy. It was successfully applied to VQ-VAE, VQ-GAN, and CVQ, improving all of them.
  • Improved Codebook Usage: The method significantly alleviates "codebook collapse" (where only a few tokens are ever used). By giving tokens semantic meaning, the model utilizes a much larger percentage of its available vocabulary.

Key Results

LG-VQ consistently outperformed baseline models in both reconstruction quality and downstream multi-modal performance:

  • Image Reconstruction: On MS-COCO, adding LG-VQ to VQ-GAN improved the Fréchet Inception Distance (FID) from 14.45 to 10.72, a significant jump in perceptual quality.
  • Text-to-Image Generation: On CelebA-HQ, the FID improved from 15.29 to 12.61, with qualitative results showing a much better grasp of specific attributes like "glasses" or "no beard."
  • Visual Question Answering (VQA): On the COCO-QA dataset, accuracy increased by approximately 8.32% compared to the VQ-GAN baseline.
  • Semantic Retrieval: The model showed a superior ability to retrieve the correct text description for a given image, demonstrating tight vision-language alignment.
Figure 4: Examples of the top-1 most similar text selected on image-to-text retrieval task.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Simplified Alignment: The current model assumes a relatively simple mapping between words and codes. In reality, one code might represent multiple words, or vice versa.
  • Performance Gap: While LG-VQ significantly improves VQ-based reasoning, its performance on tasks like image captioning still lags behind massive, end-to-end specialized models.
  • Future Directions: The team plans to investigate more complex many-to-many relationships between codes and words and explore how this framework can better interact with Large Language Models (LLMs) for even deeper multi-modal understanding.

Summary updated

Abstract

Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims to learn a codebook to encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. Although existing methods have shown superior performance, most methods prefer to learn a single-modal codebook (\emph{e.g.}, image), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (\emph{e.g.}, text-to-image, image captioning) due to the existence of modal gaps. In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to learn a codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre-trained text semantics as prior knowledge, then design two novel alignment modules (\emph{i.e.}, Semantic Alignment Module, and Relationship Alignment Module) to transfer such prior knowledge into codes for achieving codebook text alignment. In particular, our LG-VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that our method achieves superior performance on reconstruction and various multi-modal downstream tasks.
BibTeX
@inproceedings{liang2024lg,
  author = {Guotao Liang and Baoquan Zhang and Yaowei Wang and Xutao Li and Yunming Ye and Huaibin Wang and Chuyao Luo and Kola Ye and linfeng Luo},
  title = {{LG-VQ: Language-Guided Codebook Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=SYiB7P3hr2},
}
Figure