TL;DR
LG-VQ (Language-Guided Vector Quantization) is a novel framework that bridges the "modal gap" in image synthesis by aligning discrete visual codebooks with language semantics. By leveraging pre-trained text encoders and two new alignment modules, it enables VQ models to learn codebooks that are not just good at reconstructing pixels, but also rich in high-level semantic knowledge.
What Problem Does This Paper Address?
Vector Quantization (VQ) is the backbone of modern high-fidelity image synthesis (like VQ-VAE and VQ-GAN). It works by compressing images into a sequence of discrete "tokens" from a learned codebook. However, traditional VQ methods are single-modal: they focus almost entirely on low-level visual features like edges, textures, and colors.
When these codebooks are used for multi-modal tasks—such as generating an image from a text prompt or answering a question about a photo—they often struggle. This is because there is a significant "modal gap" between the visual tokens and the language used to describe them. A standard codebook might know how to draw a "racket," but it doesn't inherently understand the semantic relationship between a "woman" and a "racket" in the context of "holding."
Methods
The authors propose LG-VQ, a model-agnostic framework that can be integrated into existing VQ architectures to inject language priors into the codebook learning process.
The framework introduces three specific training objectives:
- Semantic Alignment Module (SAM):
- Global Semantic Alignment ($L_{gsa}$): Uses a contrastive loss (InfoNCE) to ensure the global representation of an image's visual codes matches the global embedding of its corresponding text description, using a pre-trained CLIP model as a teacher.
- Masked Text Prediction ($L_{mtp}$): In a "fill-in-the-blanks" task, the model must use visual codes to predict words that have been masked out of the image's caption. This forces the codebook to capture fine-grained semantic details.
- Relationship Alignment Module (RAM):
- Relationship Alignment Supervision ($L_{ras}$): This is the most novel component. It calculates the semantic similarity between pairs of words in a sentence (e.g., "girl" and "earring") and enforces that the visual codes corresponding to those words maintain a similar relationship in the visual feature space.
Datasets
The model was evaluated across four diverse public datasets to test its generality:
- TextCaps: Focuses on images containing text, requiring complex reading and reasoning.
- CelebA-HQ: High-quality human face images.
- CUB-200: Fine-grained bird species images and descriptions.
- MS-COCO: A standard large-scale benchmark for general object detection and captioning.
Key Insights and Ideas
- Beyond Pixels: The core "aha moment" is that a codebook shouldn't just be a compression tool; it should be a semantic bridge. By forcing visual tokens to "explain" text, they become much more useful for downstream reasoning.
- Relationships are Key: Simply aligning "cat" with a cat-token isn't enough. The authors show that capturing the interaction between entities (the "Relationship Alignment") is crucial for complex tasks like Visual Question Answering (VQA).
- Model-Agnosticism: LG-VQ isn't a new VQ model; it's a training strategy. It was successfully applied to VQ-VAE, VQ-GAN, and CVQ, improving all of them.
- Improved Codebook Usage: The method significantly alleviates "codebook collapse" (where only a few tokens are ever used). By giving tokens semantic meaning, the model utilizes a much larger percentage of its available vocabulary.
Key Results
LG-VQ consistently outperformed baseline models in both reconstruction quality and downstream multi-modal performance:
- Image Reconstruction: On MS-COCO, adding LG-VQ to VQ-GAN improved the Fréchet Inception Distance (FID) from 14.45 to 10.72, a significant jump in perceptual quality.
- Text-to-Image Generation: On CelebA-HQ, the FID improved from 15.29 to 12.61, with qualitative results showing a much better grasp of specific attributes like "glasses" or "no beard."
- Visual Question Answering (VQA): On the COCO-QA dataset, accuracy increased by approximately 8.32% compared to the VQ-GAN baseline.
- Semantic Retrieval: The model showed a superior ability to retrieve the correct text description for a given image, demonstrating tight vision-language alignment.
Limitations and Future Work
The authors acknowledge a few key limitations:
- Simplified Alignment: The current model assumes a relatively simple mapping between words and codes. In reality, one code might represent multiple words, or vice versa.
- Performance Gap: While LG-VQ significantly improves VQ-based reasoning, its performance on tasks like image captioning still lags behind massive, end-to-end specialized models.
- Future Directions: The team plans to investigate more complex many-to-many relationships between codes and words and explore how this framework can better interact with Large Language Models (LLMs) for even deeper multi-modal understanding.
Summary updated