Highly Compressed Tokenizer Can Generate Without Training

L. Lao Beyer, T. Li, X. Chen, Sertac Karaman, K. He

Abstract

Recent advances in image tokenization, specifically 1D tokenizers like TiTok, have achieved extreme compression ratios, representing a $256 \times 256$ image with as few as 32 discrete tokens. This paper explores the inherent generative capabilities of such highly compressed latent spaces. We demonstrate that the decoder of a 1D tokenizer acts as a powerful natural image prior, allowing for complex generative tasks—including text-guided editing, inpainting, and image synthesis—through simple test-time optimization of the discrete tokens. Remarkably, this approach requires no additional training of a generative model, yet achieves competitive performance on ImageNet, suggesting that a sufficiently expressive tokenizer can bypass the need for traditional generative training.

Introduction

The standard paradigm for modern image generation involves two distinct components: a tokenizer (an autoencoder that maps images to a compact latent space) and a generative model (such as a Diffusion model or Transformer) that learns the distribution of those latents. Training these generative models is computationally intensive and requires massive datasets.

This research investigates whether a sufficiently powerful tokenizer can eliminate the need for a separate generative model. By compressing an image into a tiny sequence of just 32 discrete tokens, the decoder must inherently "understand" realistic image structures to reconstruct them from minimal information. We explore treating this latent space as a playground for direct optimization rather than training a new model to navigate it.

Methodology: Optimization and Editing

The researchers leverage the TiTok (1D tokenizer) architecture. Unlike 2D tokenizers where tokens represent local patches, 1D tokens represent global, semantic attributes. The paper introduces two primary methods for generation and editing without training:

  1. Gradient-Based Latent Optimization: For tasks like text-to-image editing or inpainting, the authors use test-time optimization. They initialize tokens from a "seed" image or noise and update them via backpropagation to maximize an objective, such as CLIP similarity for text prompts. Gradients are passed through the discrete bottleneck using a straight-through estimator.
  2. "Copy & Paste" Token Swapping: The authors discovered that specific positions in the 32-token sequence correspond to distinct high-level attributes. By swapping a single token from a "reference" image into a "target" image, they can perform fine-grained edits without any optimization at all.
Figure 1: Overview of the test-time optimization process. A seed is encoded into 32 discrete tokens, which are then optimized to maximize a specific objective (e.g., CLIP similarity) before being decoded into the final image.

Semantic Disentanglement in 1D Latents

A key insight of this work is that 1D tokens naturally learn to represent global concepts rather than local textures. This allows for a high degree of semantic disentanglement. For instance, token #18 might control background blur, while token #31 governs scene lighting. This global view is a significant departure from 2D grid-based latents (like VQGAN), where a single token only contains information about a small spatial patch.

Figure 2: Examples of attribute editing via token swapping. By identifying tokens corresponding to specific attributes like background blur, scene lighting, or sharpening, users can "copy and paste" these characteristics from a reference image to a target image.

Key Results and Performance

Despite the lack of explicit generative training, the optimization approach yields impressive results:

  • Competitive Quality: On ImageNet, the training-free approach achieved a Frechet Inception Distance (FID) of 8.2 (using 1,000 seed images), outperforming several older, fully-trained generative models like IC-GAN (FID 15.6).
  • Efficiency: Optimization takes approximately 7 seconds per image on a single NVIDIA A100. While slower than a single GAN inference pass, it is remarkably efficient given it requires zero training data for the generative task.
  • The Decoder as a Prior: The results suggest that as compression ratios increase, the decoder's role shifts from simple reconstruction to active generation, effectively serving as a strong "natural image prior."

Limitations and Future Outlook

While powerful, the method currently relies on "seed" images for the best results, as starting from pure noise can be challenging for complex layouts. Additionally, while global attributes are easily manipulated, complex semantic changes—such as altering the specific pose of a subject—remain difficult through simple token swapping. Future work will likely explore even higher compression ratios (e.g., 8 or 16 tokens), which may further enhance the generative power of the decoder and move us closer to truly training-free high-resolution synthesis.

Summary updated

Abstract

Commonly used image tokenizers produce a 2D grid of spatially arranged tokens. In contrast, so-called 1D image tokenizers represent images as highly compressed one-dimensional sequences of as few as 32 discrete tokens. We find that the high degree of compression achieved by a 1D tokenizer with vector quantization enables image editing and generative capabilities through heuristic manipulation of tokens, demonstrating that even very crude manipulations -- such as copying and replacing tokens between latent representations of images -- enable fine-grained image editing by transferring appearance and semantic attributes. Motivated by the expressivity of the 1D tokenizer's latent space, we construct an image generation pipeline leveraging gradient-based test-time optimization of tokens with plug-and-play loss functions such as reconstruction or CLIP similarity. Our approach is demonstrated for inpainting and text-guided image editing use cases, and can generate diverse and realistic samples without requiring training of any generative model.
BibTeX
@inproceedings{beyer2025highly,
  author = {L. Lao Beyer and T. Li and X. Chen and Sertac Karaman and K. He},
  title = {{Highly Compressed Tokenizer Can Generate Without Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=rcjgzbYvhF},
}
Figure