Exploration Into Translation-Equivariant Image Quantization

Woncheol Shin, Gyubok Lee, Jiyoung Lee, Eunyi Lyou, Joonseok Lee, Edward Choi

Abstract

This exploratory study discovers that current image quantization (vector quantization) methods do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on complex anti-aliasing filters, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, (2) image-to-text generation, and (3) sample efficiency analysis. Our results empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also accuracy over VQGAN by up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.

The Challenge of Translation Equivariance in VQGANs

In modern generative AI, models like DALL-E do not process raw pixels directly. Instead, they utilize Vector Quantization (VQ) to transform an image into a grid of discrete tokens. A fundamental expectation for these models is translation equivariance: if an object in an image shifts by a few pixels, its corresponding discrete tokens should simply shift by the same amount in the grid representation.

However, standard VQGANs frequently fail this test. The root cause is aliasing—a signal processing artifact resulting from downsampling operations like strided convolutions. A minor spatial shift in the input image can introduce enough "noise" in the feature space to push a feature vector closer to an entirely different codebook entry. This inconsistency forces downstream models, such as Transformers, to learn redundant or conflicting rules for the same semantic object at different positions.

Achieving Equivariance through Orthogonality

While traditional signal processing fixes involve complex anti-aliasing filters, this research proposes a geometric alternative: TE-VQGAN (Translation-Equivariant VQGAN).

Figure 1: Illustration of how aliasing causes inconsistent token mapping (a vs b) and how enforcing orthogonality (c) stabilizes the representation.

As shown in Figure 1, the core idea is to make the model robust to aliasing noise rather than eliminating it. By adding a regularization term that enforces orthogonality among codebook embeddings, the model ensures that vectors are as far apart as possible in the feature space. This creates a "buffer zone" where the jitter caused by aliasing is unlikely to push a feature vector into the territory of a different codebook index. Consequently, the mapping remains stable even when the input is translated.

Controlled Experimental Framework

To isolate the effects of translation equivariance, the authors developed a strictly controlled environment using MNIST-based image-caption pairs. The dataset consists of 300,000 samples where 1–4 digits are placed in various quadrants of a 64x64 canvas, accompanied by descriptive captions.

Figure 3: Example of the synthetic dataset featuring multiple digits with spatial and color-based captions.

This setup allowed for testing zero-shot generalization. For instance, certain combinations (like a "white 0 in the upper left") were excluded from the training set to see if the model could generalize spatial concepts it had learned from other digit/position combinations.

Performance Gains and Sample Efficiency

The empirical results demonstrate that translation equivariance acts as a significant performance multiplier:

  • Generative Accuracy: TE-VQGAN improved digit accuracy in text-to-image (T2I) tasks by up to +11.9% and image-to-text (I2T) tasks by +3.9% compared to standard VQGAN.
  • Sample Efficiency: A TE-VQGAN trained on only 100,000 samples outperformed a standard VQGAN trained on 300,000 samples. Consistent tokenization allows the model to learn the underlying "rules" of the dataset with far less data.
  • Semantic vs. Pixel Fidelity: Interestingly, enforcing orthogonality slightly increases reconstruction loss (the images look marginally less sharp). However, this trade-off vastly improves semantic accuracy, which is often more critical for downstream reasoning tasks.

Conclusion and Future Outlook

This study highlights that the structure of the VQ codebook itself is a primary bottleneck for spatial consistency. By prioritizing geometric orthogonality, TE-VQGAN achieves superior translation equivariance and sample efficiency without the overhead of complex filtering. While these results are currently demonstrated on synthetic MNIST-style data, they provide a compelling roadmap for improving the robustness of large-scale generative models on complex, real-world datasets like ImageNet.

Summary updated

Abstract

This is an exploratory study that discovers the current image quantization (vector quantization) do not satisfy translation equivariance in the quantized space due to aliasing. Instead of focusing on anti-aliasing, we propose a simple yet effective way to achieve translation-equivariant image quantization by enforcing orthogonality among the codebook embeddings. To explore the advantages of translation-equivariant image quantization, we conduct three proof-of-concept experiments with a carefully controlled dataset: (1) text-to-image generation, where the quantized image indices are the target to predict, (2) image-to-text generation, where the quantized image indices are given as a condition, (3) using a smaller training set to analyze sample efficiency. From the strictly controlled experiments, we empirically verify that the translation-equivariant image quantizer improves not only sample efficiency but also the accuracy over VQGAN up to +11.9% in text-to-image generation and +3.9% in image-to-text generation.
BibTeX
@inproceedings{shin2023exploration,
  author = {Woncheol Shin and Gyubok Lee and Jiyoung Lee and Eunyi Lyou and Joonseok Lee and Edward Choi},
  title = {{Exploration Into Translation-Equivariant Image Quantization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096052},
}
Figure