Learning Visual Generative Priors without Text

Shuailei Ma, Kecheng Zheng, Ying Wei, Wei Wu, Fan Lu, Yifei Zhang, Chen-Wei Xie, Biao Gong, Jiapeng Zhu, Yujun Shen

Abstract

Modern generative models predominantly rely on text-to-image (T2I) pre-training to establish visual priors. However, this approach is constrained by the scarcity and cost of high-quality image-text pairs and the complexity of simultaneously learning texture modeling and cross-modality alignment. We present Lumos, a framework that shifts the generative foundation to a self-supervised image-to-image (I2I) task. By pre-training on 190 million unannotated images, Lumos develops a robust visual prior focused on structural and textural fidelity. Our results demonstrate that this pure vision-based prior not only matches or exceeds T2I-based models in image synthesis but also provides a superior foundation for downstream tasks such as 3D novel view synthesis and video generation, requiring significantly less labeled data.

Introduction

The current landscape of generative AI is dominated by models like Stable Diffusion, which utilize text-to-image (T2I) objectives as their primary visual prior. While effective, this paradigm introduces a significant bottleneck: the reliance on massive, curated datasets of image-text pairs. Scaling these datasets is labor-intensive and expensive. Furthermore, the T2I objective forces a model to solve two distinct problems at once—learning the visual characteristics of the world (texture modeling) and learning how language maps to those pixels (cross-modality alignment).

Lumos challenges this status quo by decoupling these tasks. By focusing initially on an Image-to-Image (I2I) reconstruction task, the model can leverage billions of unannotated images available in the wild. This approach allows the model to master the "how" of visual generation—lighting, geometry, and texture—before being introduced to the "what" of linguistic descriptions.

Methodology: The I2I Prior

The core of the Lumos framework is a self-supervised pre-training phase designed to build a foundational visual prior without text.

Pre-training Architecture

Lumos utilizes a Diffusion Transformer (DiT) architecture. During pre-training, the model is tasked with reconstructing an image based on visual features extracted from a frozen vision encoder. Specifically, the authors found that using DINO-B as the encoder provided a richer signal for reconstruction compared to text-aligned encoders like CLIP. This I2I task forces the transformer to learn intricate texture modeling and structural consistency across a diverse dataset of 190 million unannotated images, sourced from collections like LAION-5B, SAM, and ImageNet-1K.

Transfer Learning and Adaptation

Once the I2I prior is established, it serves as a versatile foundation for several downstream generative tasks:

  1. Text-to-Image (T2I): The model is fine-tuned by replacing the image encoder with a text encoder (T5). Remarkably, this phase requires 90% less labeled data than training a T2I model from scratch.
  2. Novel View Synthesis (NVS): For 3D tasks, the model is conditioned on an input image and a relative camera viewpoint, allowing it to generate consistent views of an object from new angles.
  3. Image-to-Video (I2V): By adding temporal blocks to the transformer, Lumos can generate continuous video frames from a single reference image, maintaining temporal coherence through its strong structural prior.

Experimental Results

Lumos demonstrates state-of-the-art performance across multiple benchmarks, proving that text-free pre-training is a highly efficient path to generative excellence.

Efficiency in Text-to-Image Synthesis

When fine-tuned for T2I tasks, Lumos achieved an FID of 6.44 on the MS-COCO benchmark. This outperforms established models like PixArt-α (7.32), despite Lumos using significantly fewer text-image pairs during the fine-tuning stage. This suggests that the I2I prior makes the model more robust and less sensitive to noisy text data.

Superiority in 3D and Video Generation

Lumos excels in tasks where text prompts are often insufficient or secondary to visual conditioning:

  • 3D Synthesis: In Novel View Synthesis (NVS), Lumos-NVS achieved a PSNR of 19.63 on the Google Scanned Objects (GSO) dataset, significantly outperforming Zero123 (17.73) and showing much higher structural consistency (SSIM).
  • Video Generation: On the UCF-101 benchmark, Lumos-I2V achieved a Frechet Video Distance (FVD) of 399.59, a substantial improvement over the 532.77 FVD recorded by models relying on standard T2I priors.

Discussion and Insights

The "Late Bloomer" Effect

A key insight from the research is the performance trajectory of different encoders. While CLIP-based encoders (which are text-aligned) show better performance in the early stages of training, DINO-based encoders (pure vision) eventually achieve superior image quality. This suggests that pure vision features provide a more detailed and nuanced signal for high-fidelity reconstruction over long-term training.

Scalability

The research confirms that generative performance scales predictably with unannotated data. Increasing the pre-training set from 10 million to 190 million images consistently improved downstream task quality. This validates the use of unannotated web-scale data as a viable and cost-effective alternative to labeled datasets.

Conclusion and Future Work

Lumos demonstrates that text-to-image alignment is not a prerequisite for a powerful visual generative prior. By prioritizing texture modeling through self-supervised I2I pre-training, Lumos achieves superior efficiency and performance in 2D, 3D, and video domains. Future research will explore scaling this approach to billions of images and investigating the integration of these pure visual priors into multimodal large language models (MLLMs) to enhance their visual synthesis capabilities.

Summary updated

Abstract

Although text-to-image (T2I) models have recently thrived as visual generative priors, their reliance on high-quality text-image pairs makes scaling up expensive. We argue that grasping the cross-modality alignment is not a necessity for a sound visual generative prior, whose focus should be on texture modeling. Such a philosophy inspires us to study image-to-image (I2I) generation, where models can learn from in-the-wild images in a self-supervised manner. We first develop a pure vision-based training framework, Lumos, and confirm the feasibility and the scalability of learning I2I models. We then find that, as an upstream task of T2I, our I2I model serves as a more foundational visual prior and achieves on-par or better performance than existing T2I models using only 1/10 text-image pairs for fine-tuning. We further demonstrate the superiority of I2I priors over T2I priors on some text-irrelevant visual generative tasks, like image-to-3D and image-to-video. Our project page is available at https://ant-research.github.io/lumos.
BibTeX
@inproceedings{ma2025learning,
  author = {Shuailei Ma and Kecheng Zheng and Ying Wei and Wei Wu and Fan Lu and Yifei Zhang and Chen-Wei Xie and Biao Gong and Jiapeng Zhu and Yujun Shen},
  title = {{Learning Visual Generative Priors without Text}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00754},
}
Figure