Abstract
This is an exploratory study that discovers the current image quantization (vector quantization) do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on anti-aliasing, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, where the quantized image indices are the target to predict, (2) image-to-text generation, where the quantized image indices are given as a condition, (3) using a smaller training set to analyze sample efficiency. From the strictly controlled experiments, we empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also the accuracy over VQGAN up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.
The Problem: Broken Translation Equivariance
In generative AI, models like DALL-E utilize Vector Quantization (VQ) to convert continuous images into discrete tokens. Ideally, these models should exhibit translation equivariance: if an object moves within an image, its token representation should remain identical, merely shifting its position in the grid.
However, standard quantizers like VQGAN often fail this test due to aliasing—noise introduced by downsampling operations. As illustrated in the figure below, a digit '5' in the top-left might be mapped to one codebook entry, while the same digit shifted to the top-right might be mapped to an entirely different entry because the feature vector drifts into a different "territory" in the embedding space.
Proposed Method: TE-VQGAN
Rather than attempting to eliminate aliasing noise entirely through complex filters, this paper proposes TE-VQGAN. The core idea is to make the quantization step robust to noise by enforcing orthogonality among codebook embeddings.
By forcing all codebook vectors to be perpendicular to one another, the model creates a significant "safety margin." Even if aliasing noise $\epsilon$ shifts a feature vector, the vector is far more likely to remain closest to its original codebook entry. This simple regularization ensures that the "essence" of an object remains consistent regardless of its spatial coordinates.
Experimental Setup and Datasets
To rigorously evaluate this approach, the authors developed a controlled MNIST-based dataset consisting of 64x64 images with one to four digits. Captions describe the color, digit, and location (e.g., "the red 3 is on the upper right").
A critical component of the study was the use of Zero-Shot Constraints. The researchers intentionally withheld specific combinations from the training set—such as a "white 0" in the "upper left"—to test if the model could generalize the underlying logic of composition rather than just memorizing layouts.
Key Results
TE-VQGAN demonstrated significant improvements over standard VQGAN and other anti-aliasing baselines:
- Performance Gains: TE-VQGAN improved digit accuracy by +11.9% in text-to-image tasks and +3.9% in image-to-text tasks.
- Sample Efficiency: When the training set was reduced from 300K to 100K samples, TE-VQGAN’s performance remained stable (0.931 to 0.919), while standard VQGAN saw a sharp decline (0.812 to 0.739).
- Codebook Utilization: Interestingly, TE-VQGAN used far fewer entries (14 vs. 252 for standard VQGAN), suggesting it successfully learned a minimal set of discrete concepts.
Conclusion and Future Work
While TE-VQGAN significantly boosts robustness and generalization, it introduces a trade-off in image fidelity. The strict orthogonality and reduced codebook usage lead to higher reconstruction errors compared to vanilla VQGAN. Future research will focus on scaling this approach to complex, real-world datasets like ImageNet, where balancing visual detail with conceptual robustness remains a primary challenge.
Summary updated