Introduction
Latent Diffusion Models (LDMs)—the architecture behind models like Stable Diffusion and FLUX—rely on visual tokenizers, typically Variational Autoencoders (VAEs), to compress raw pixels into a manageable latent space. Traditionally, these tokenizers are trained from scratch to minimize reconstruction loss. This creates a fundamental asymmetry: while the decoder learns to recreate pixels perfectly, the encoder’s latent space often becomes disorganized, prioritizing low-level details over high-level semantics. This lack of structure results in poor "diffusability," making it harder for the diffusion model to learn the data distribution.
While recent approaches have attempted to regularize these spaces using semantic losses, the encoders still learn high-level concepts from scratch. AlignTok proposes a paradigm shift: instead of training image tokenizers from scratch, it repurposes pretrained foundation models (such as DINOv2) as encoders for latent diffusion.
The AlignTok Framework
To transform a visual foundation encoder into a diffusion-ready tokenizer, the authors introduce a three-stage alignment process:
- Latent Alignment: The pretrained foundation encoder is frozen. A lightweight adapter and a decoder are trained to map the encoder’s high-dimensional semantic features into a compact latent space and reconstruct the original image. This establishes a strong "semantic anchor."
- Perceptual Alignment: All components (encoder, adapter, and decoder) are unfrozen and trained jointly. To prevent the encoder from forgetting its rich semantics while learning fine-grained textures, a novel Semantic Preservation Loss is introduced. This ensures the new latent codes remain close to the original semantic codes established in Stage 1.
- Decoder Refinement: Finally, the encoder and adapter are frozen once more, and only the decoder is fine-tuned. This maximizes reconstruction fidelity without disrupting the optimized latent space.
Key Insights
- Semantics Precede Pixels: The core philosophy of AlignTok is that learning high-level semantics is significantly harder than learning pixel reconstruction. By starting with a foundation model, the most challenging aspect of representation learning is already resolved.
- Enhanced Diffusability: A semantically grounded latent space is much easier for a diffusion model to navigate. Because the latent space is organized by meaning rather than just pixel clusters, the diffusion model converges much faster.
- Robustness to Guidance Scales: Models trained with AlignTok perform remarkably well even at low Classifier-Free Guidance (CFG) scales. This suggests the latent space already encodes well-separated class semantics, requiring less artificial "pushing" to achieve high fidelity.
- Optimal Backbone: After testing various architectures (MAE, SigLIP 2, DINOv2), the authors found that DINOv2 provided the best balance between reconstruction ability and generative performance.
Experimental Results
The researchers evaluated AlignTok across multiple benchmarks, including ImageNet (256×256) for class-conditional generation and a LAION-2B subset for large-scale text-to-image (T2I) generation.
- Accelerated Convergence: On ImageNet, the model reached a highly competitive generation FID (gFID) of 1.90 in just 64 epochs—roughly five times faster than the state-of-the-art VA-VAE.
- Superior Generation Quality: When scaled to LAION, AlignTok consistently outperformed heavyweights like the FLUX VAE and VA-VAE across multiple metrics, including HPSv2 (human preference), PickScore, and Aesthetic Scores.
- Training Efficiency: Despite utilizing a larger encoder, the overall training pipeline is more efficient. The cumulative GPU hours required for AlignTok are lower than training a standard VAE from scratch because the alignment phase converges rapidly.
Limitations and Future Work
While AlignTok significantly improves generated image quality, its pure reconstruction fidelity (copying an input image exactly) still lags slightly behind models like the FLUX VAE. The authors suggest this gap could be closed with larger decoders or extended training. Additionally, current evaluations were limited to 512×512 resolution; scaling to megapixel resolutions remains a logical next step. Finally, this alignment paradigm offers a promising blueprint for other modalities, potentially extending to video tokenization or discrete tokenizers for autoregressive models.