Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework

A Vetrov Vladimir, Viacheslav Vasilev, Andrei V. Filatov, И. С. Павлов, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Bonnie B. Anton, Konstantin Kulikov, Andrey Kuznetsov, Denis Dimitrov

Introduction

The field of text-to-image (T2I) synthesis has seen rapid advancement, yet significant challenges remain regarding architectural complexity and inference efficiency. Many state-of-the-art models require specialized architectures or complex "hacks" to perform auxiliary tasks like inpainting or video generation. Furthermore, the high computational cost of diffusion processes often limits real-time accessibility. Kandinsky 3 addresses these hurdles by introducing a simplified yet powerful 11.9-billion parameter latent diffusion framework designed for both high-fidelity image synthesis and seamless adaptation to diverse generative tasks.

Architectural Design

Kandinsky 3 is built upon a latent diffusion foundation comprising three core components: a text encoder, a denoising U-Net, and an image decoder.

Text Encoding and Language Understanding

Unlike many models that rely on smaller CLIP-based encoders, Kandinsky 3 utilizes Flan-UL2, a massive 8.6B parameter encoder. This choice significantly enhances the model's "common sense" understanding and its ability to interpret complex, nuanced prompts.

Denoising U-Net

The backbone of the generative process is a 3.0B parameter U-Net. The researchers implemented modified BigGAN-deep blocks, discovering that increasing network depth through bottleneck structures (similar to ResNet-50) was more effective than increasing layer width. This approach allowed for a doubling of layers without an increase in total parameter count, improving training stability and feature extraction. The architecture employs a hybrid layering strategy: purely convolutional blocks handle initial high-resolution layers to process local textures, while Transformer layers are introduced at lower resolutions to manage global structural interactions.

Image Reconstruction

For the final stage, the Sber-MoVQGAN (0.27B parameters) serves as the decoder, reconstructing high-quality images from the latent space.

Training Methodology and Data

The model was trained on a curated dataset of approximately 150 million text-image pairs, sourced from LAION-5B, COYO-700M, and internal web-crawled data. Rigorous filtering was applied to ensure aesthetic quality and remove watermarks. Training proceeded in stages, beginning at 256x256 resolution and culminating in a mixed-resolution stage supporting outputs up to 1024x1024.

Key Innovations: Distillation and Refinement

To address inference speed, the authors developed a distilled version of Kandinsky 3. This version can generate images in just 4 steps—roughly 3x faster than the base model. A unique "Refiner" strategy was introduced where the base model establishes the initial logic and structure of the image, while the distilled model performs the final denoising steps. This hybrid approach leverages the base model's superior text alignment and the distilled model's enhanced aesthetic output.

Additionally, the framework integrates Neural-Chat-7b-v3-1, an LLM used for "prompt beautification." This component automatically expands short or vague user queries into descriptive prompts, improving the final visual result without requiring expert prompt engineering from the user.

Multifunctional Capabilities

Kandinsky 3 is designed as a versatile framework. By incorporating IP-Adapters and ControlNet, the model supports:

  • Image Fusion and Variations: Combining distinct visual elements (e.g., a cat and a boot).
  • Inpainting and Outpainting: Seamlessly editing or extending existing images.
  • Video Generation: Adapting the T2I weights for text-to-video and image-to-video tasks.

Results and Evaluation

In side-by-side human evaluations, Kandinsky 3 (enhanced by prompt beautification) performed comparably to or better than industry leaders such as SDXL, DALL-E 3, and Midjourney 5.2. While the distillation process involves a minor trade-off in precise text comprehension, it significantly boosts visual aesthetics and inference speed, maintaining competitive scores on the DrawBench benchmark.

Conclusion and Future Work

Kandinsky 3 represents a significant step toward a unified, efficient generative framework. While current limitations include lower resolutions for video tasks (512x512) and slight nuances in distilled text comprehension, the model's open-source release provides a robust foundation for community-driven optimization. By prioritizing depth over width and integrating LLM-based prompt enhancement, Kandinsky 3 achieves a balance of speed, versatility, and high-fidelity synthesis.

Summary updated

BibTeX
@misc{vladimir2024kandinsky,
  author = {A Vetrov Vladimir and Viacheslav Vasilev and Andrei V. Filatov and И. С. Павлов and Julia Agafonova and Nikolai Gerasimenko and Anna Averchenkova and Evelina Mironova and Bonnie B. Anton and Konstantin Kulikov and Andrey Kuznetsov and Denis Dimitrov},
  title = {{Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework}},
  year = {2024},
  eprint = {2410.21061},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21061},
}
Figure