Abstract
Modern generative models predominantly rely on text-to-image (T2I) pre-training to establish visual priors. However, this approach is constrained by the scarcity and cost of high-quality image-text pairs and the complexity of simultaneously learning texture modeling and cross-modality alignment. We present Lumos, a framework that shifts the generative foundation to a self-supervised image-to-image (I2I) task. By pre-training on 190 million unannotated images, Lumos develops a robust visual prior focused on structural and textural fidelity. Our results demonstrate that this pure vision-based prior not only matches or exceeds T2I-based models in image synthesis but also provides a superior foundation for downstream tasks such as 3D novel view synthesis and video generation, requiring significantly less labeled data.
Introduction
The current landscape of generative AI is dominated by models like Stable Diffusion, which utilize text-to-image (T2I) objectives as their primary visual prior. While effective, this paradigm introduces a significant bottleneck: the reliance on massive, curated datasets of image-text pairs. Scaling these datasets is labor-intensive and expensive. Furthermore, the T2I objective forces a model to solve two distinct problems at once—learning the visual characteristics of the world (texture modeling) and learning how language maps to those pixels (cross-modality alignment).
Lumos challenges this status quo by decoupling these tasks. By focusing initially on an Image-to-Image (I2I) reconstruction task, the model can leverage billions of unannotated images available in the wild. This approach allows the model to master the "how" of visual generation—lighting, geometry, and texture—before being introduced to the "what" of linguistic descriptions.
Methodology: The I2I Prior
The core of the Lumos framework is a self-supervised pre-training phase designed to build a foundational visual prior without text.
Pre-training Architecture
Lumos utilizes a Diffusion Transformer (DiT) architecture. During pre-training, the model is tasked with reconstructing an image based on visual features extracted from a frozen vision encoder. Specifically, the authors found that using DINO-B as the encoder provided a richer signal for reconstruction compared to text-aligned encoders like CLIP. This I2I task forces the transformer to learn intricate texture modeling and structural consistency across a diverse dataset of 190 million unannotated images, sourced from collections like LAION-5B, SAM, and ImageNet-1K.
Transfer Learning and Adaptation
Once the I2I prior is established, it serves as a versatile foundation for several downstream generative tasks:
- Text-to-Image (T2I): The model is fine-tuned by replacing the image encoder with a text encoder (T5). Remarkably, this phase requires 90% less labeled data than training a T2I model from scratch.
- Novel View Synthesis (NVS): For 3D tasks, the model is conditioned on an input image and a relative camera viewpoint, allowing it to generate consistent views of an object from new angles.
- Image-to-Video (I2V): By adding temporal blocks to the transformer, Lumos can generate continuous video frames from a single reference image, maintaining temporal coherence through its strong structural prior.
Experimental Results
Lumos demonstrates state-of-the-art performance across multiple benchmarks, proving that text-free pre-training is a highly efficient path to generative excellence.
Efficiency in Text-to-Image Synthesis
When fine-tuned for T2I tasks, Lumos achieved an FID of 6.44 on the MS-COCO benchmark. This outperforms established models like PixArt-α (7.32), despite Lumos using significantly fewer text-image pairs during the fine-tuning stage. This suggests that the I2I prior makes the model more robust and less sensitive to noisy text data.
Superiority in 3D and Video Generation
Lumos excels in tasks where text prompts are often insufficient or secondary to visual conditioning:
- 3D Synthesis: In Novel View Synthesis (NVS), Lumos-NVS achieved a PSNR of 19.63 on the Google Scanned Objects (GSO) dataset, significantly outperforming Zero123 (17.73) and showing much higher structural consistency (SSIM).
- Video Generation: On the UCF-101 benchmark, Lumos-I2V achieved a Frechet Video Distance (FVD) of 399.59, a substantial improvement over the 532.77 FVD recorded by models relying on standard T2I priors.
Discussion and Insights
The "Late Bloomer" Effect
A key insight from the research is the performance trajectory of different encoders. While CLIP-based encoders (which are text-aligned) show better performance in the early stages of training, DINO-based encoders (pure vision) eventually achieve superior image quality. This suggests that pure vision features provide a more detailed and nuanced signal for high-fidelity reconstruction over long-term training.
Scalability
The research confirms that generative performance scales predictably with unannotated data. Increasing the pre-training set from 10 million to 190 million images consistently improved downstream task quality. This validates the use of unannotated web-scale data as a viable and cost-effective alternative to labeled datasets.
Conclusion and Future Work
Lumos demonstrates that text-to-image alignment is not a prerequisite for a powerful visual generative prior. By prioritizing texture modeling through self-supervised I2I pre-training, Lumos achieves superior efficiency and performance in 2D, 3D, and video domains. Future research will explore scaling this approach to billions of images and investigating the integration of these pure visual priors into multimodal large language models (MLLMs) to enhance their visual synthesis capabilities.
Summary updated