Learning Visual Generative Priors without Text

Shuailei Ma, Kecheng Zheng, Ying Wei, Wei Wu, Fan Lu, Yifei Zhang, Chen-Wei Xie, Biao Gong, Jiapeng Zhu, Yujun Shen

Introduction

Modern visual generative models, such as Stable Diffusion and DALL-E, have revolutionized image synthesis by acting as powerful "visual priors" that understand the structure and appearance of the world. However, these models are almost universally trained using Text-to-Image (T2I) objectives, which couple visual understanding with linguistic alignment from the start. Researchers from Ant Group and Northeastern University have introduced Lumos, a framework that challenges this paradigm by learning visual generative priors through pure image-to-image (I2I) tasks, decoupling texture modeling from language alignment.

The Limitations of Text-to-Image Priors

The authors identify two critical bottlenecks in the current T2I-centric approach to generative modeling:

  1. Data Scalability: T2I models rely on high-quality, manually or synthetically captioned image-text pairs. While billions of unannotated images exist "in the wild," the pool of accurately captioned data is significantly smaller and more expensive to scale.
  2. Conflicting Objectives: Training a model to simultaneously master texture modeling (what things look like) and cross-modality alignment (how words map to pixels) can be counterproductive. Noisy or low-quality text descriptions often degrade the model's ability to learn intricate visual textures.

Lumos posits that a true visual prior should focus on the "how" of image construction—texture and structure—before introducing the "what" of language.

The Lumos Framework: Vision-First Pre-training

Lumos is built upon a Diffusion Transformer (DiT) backbone and utilizes a self-supervised Image-to-Image (I2I) pre-training phase. Instead of text prompts, the model is conditioned on visual features extracted from an input image by a frozen vision encoder (such as DINO-B). This allows the model to focus exclusively on reconstructing and generating high-fidelity textures from visual cues.

During pre-training, Lumos utilized a massive dataset of 190 million unannotated images filtered for high aesthetic quality. This pure vision-based approach allows the model to act as a "late bloomer." While text-aligned encoders like CLIP help models learn faster initially, pure vision encoders like DINO eventually achieve superior image quality (FID), providing a more robust foundation for generative tasks.

Versatile Downstream Transfer

Once the I2I prior is established, Lumos can be efficiently transferred to various downstream tasks with minimal additional data. The authors demonstrated this versatility across three primary domains:

  • Text-to-Image (T2I) Generation: By adding a text encoder (T5) and fine-tuning with cross-attention, Lumos masters language alignment with 90% less captioned data than traditional models.
  • Novel View Synthesis (NVS): The model can generate different angles of an object from a single reference image and a relative viewpoint, outperforming specialized models like Zero123.
  • Image-to-Video (I2V): By incorporating temporal blocks, Lumos generates continuous, high-quality video frames from a static starting image.
Lumos demonstrates strong performance across Text-to-Image (T2I), Novel View Synthesis (NVS), and Image-to-Video (I2V) tasks.

Results and Performance

The results indicate that mastering texture through I2I pre-training makes subsequent tasks significantly easier. Lumos-T2I achieved performance comparable to or better than state-of-the-art models while using only 1/10th of the text-image pairs typically required. On the MS-COCO benchmark, Lumos-I2I achieved an FID of 4.82, a massive improvement over the previous state-of-the-art I2I model, RCG (FID 12.70).

High-fidelity generation showcasing the model's ability to master complex textures and structures.

In tasks where text is often an "awkward middleman," such as 3D reconstruction and video interpolation, Lumos's vision-first approach proved naturally superior. It achieved significantly lower Fréchet Video Distance (FVD) in video generation and higher PSNR/SSIM metrics in novel view synthesis compared to T2I-prior-based models.

Conclusion

Lumos represents a shift toward more scalable and efficient foundational models for image synthesis. By demonstrating that language alignment is a downstream task rather than a pre-training requirement, the authors have opened a path toward utilizing the trillions of unannotated images available online. Future work may involve scaling this I2I pre-training even further, potentially creating a universal generative engine where text, depth, and motion are simply different control signals for a single, powerful visual prior.

Summary updated

BibTeX
@inproceedings{ma2025learning,
  author = {Shuailei Ma and Kecheng Zheng and Ying Wei and Wei Wu and Fan Lu and Yifei Zhang and Chen-Wei Xie and Biao Gong and Jiapeng Zhu and Yujun Shen},
  title = {{Learning Visual Generative Priors without Text}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00754},
}
Figure