Translation-equivariant Image Quantizer for Bi-directional Image-Text Generation

Woncheol Shin, Gyubok Lee, Jiyoung Lee, Joonseok Lee, Edward Choi

Abstract

This is an exploratory study that discovers the current image quantization (vector quantization) do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on anti-aliasing, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, where the quantized image indices are the target to predict, (2) image-to-text generation, where the quantized image indices are given as a condition, (3) using a smaller training set to analyze sample efficiency. From the strictly controlled experiments, we empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also the accuracy over VQGAN up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.

The Problem: Broken Translation Equivariance

In generative AI, models like DALL-E utilize Vector Quantization (VQ) to convert continuous images into discrete tokens. Ideally, these models should exhibit translation equivariance: if an object moves within an image, its token representation should remain identical, merely shifting its position in the grid.

However, standard quantizers like VQGAN often fail this test due to aliasing—noise introduced by downsampling operations. As illustrated in the figure below, a digit '5' in the top-left might be mapped to one codebook entry, while the same digit shifted to the top-right might be mapped to an entirely different entry because the feature vector drifts into a different "territory" in the embedding space.

Figure 1: Translation equivariance failure in standard VQGAN (a vs b) where a shifted digit maps to a different codebook entry, and the proposed fix (c) using orthogonal embeddings.

Proposed Method: TE-VQGAN

Rather than attempting to eliminate aliasing noise entirely through complex filters, this paper proposes TE-VQGAN. The core idea is to make the quantization step robust to noise by enforcing orthogonality among codebook embeddings.

By forcing all codebook vectors to be perpendicular to one another, the model creates a significant "safety margin." Even if aliasing noise $\epsilon$ shifts a feature vector, the vector is far more likely to remain closest to its original codebook entry. This simple regularization ensures that the "essence" of an object remains consistent regardless of its spatial coordinates.

Experimental Setup and Datasets

To rigorously evaluate this approach, the authors developed a controlled MNIST-based dataset consisting of 64x64 images with one to four digits. Captions describe the color, digit, and location (e.g., "the red 3 is on the upper right").

A critical component of the study was the use of Zero-Shot Constraints. The researchers intentionally withheld specific combinations from the training set—such as a "white 0" in the "upper left"—to test if the model could generalize the underlying logic of composition rather than just memorizing layouts.

Figure 4: Systematic train/test splits for the MNIST-based dataset, designed to evaluate zero-shot generalization across colors, labels, and locations.

Key Results

TE-VQGAN demonstrated significant improvements over standard VQGAN and other anti-aliasing baselines:

  • Performance Gains: TE-VQGAN improved digit accuracy by +11.9% in text-to-image tasks and +3.9% in image-to-text tasks.
  • Sample Efficiency: When the training set was reduced from 300K to 100K samples, TE-VQGAN’s performance remained stable (0.931 to 0.919), while standard VQGAN saw a sharp decline (0.812 to 0.739).
  • Codebook Utilization: Interestingly, TE-VQGAN used far fewer entries (14 vs. 252 for standard VQGAN), suggesting it successfully learned a minimal set of discrete concepts.

Conclusion and Future Work

While TE-VQGAN significantly boosts robustness and generalization, it introduces a trade-off in image fidelity. The strict orthogonality and reduced codebook usage lead to higher reconstruction errors compared to vanilla VQGAN. Future research will focus on scaling this approach to complex, real-world datasets like ImageNet, where balancing visual detail with conceptual robustness remains a primary challenge.

Summary updated

Abstract

This is an exploratory study that discovers the current image quantization (vector quantization) do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on anti-aliasing, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, where the quantized image indices are the target to predict, (2) image-to-text generation, where the quantized image indices are given as a condition, (3) using a smaller training set to analyze sample efficiency. From the strictly controlled experiments, we empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also the accuracy over VQGAN up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.
BibTeX
@inproceedings{shin2023translation,
  author = {Woncheol Shin and Gyubok Lee and Jiyoung Lee and Joonseok Lee and Edward Choi},
  title = {{Translation-equivariant Image Quantizer for Bi-directional Image-Text Generation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096052},
}
Figure