TL;DR
DreamOmni is a unified generative model that seamlessly integrates text-to-image (T2I) generation with a wide array of image editing tasks—such as drag-based editing, inpainting, and subject-driven generation—within a single framework. By leveraging a novel synthetic collage data pipeline and a Vision-Language Model (VLM) backbone, it achieves state-of-the-art performance while significantly simplifying the deployment of multi-task vision systems.
What Problem Does This Paper Address?
While text-to-image models have reached incredible heights, they remain fragmented. If you want to edit an image, you typically need specialized "plugins" like ControlNet, IP-Adapter, or models with modified input channels like InstructP2P. This reliance on specialized frameworks makes multi-tasking difficult and deployment cumbersome.
Furthermore, training these models requires high-quality "paired" data (e.g., an image before and after a specific edit). Such data is notoriously difficult to collect or synthesize accurately; manual labeling is slow, and automated methods often introduce artifacts or fail to follow complex instructions. DreamOmni seeks to bridge this gap by creating a native, unified model trained on a massive, programmatically generated synthetic dataset.
Methods
The authors propose a unified architecture and a scalable data generation strategy:
- The Framework: DreamOmni uses a 2.5B parameter Diffusion Transformer (DiT) as its backbone. Unlike traditional models that use CLIP or T5 for text, DreamOmni adopts a Vision-Language Model (VLM)—specifically Qwen2-VL—as its encoder. This allows the model to process text instructions and reference images in a unified latent space.
- Architectural Refinements: To improve training speed, the authors incorporated UNet-style residual connections into the DiT blocks. They discovered that concentrating computations on $2\times$ downsampled latents (rather than spreading them across multiple scales) provides the best trade-off between performance and efficiency.
- Synthetic Collage Pipeline: To solve the data scarcity problem, the researchers developed a pipeline that uses "sticker-like" elements. By programmatically placing, scaling, rotating, and replacing these elements on a canvas, they can generate billions of perfectly paired training examples for tasks like "add a hat" or "move the cat to the left."
Datasets
DreamOmni was trained on a massive scale:
- Real-world Data: 125M images, including the LAION dataset (103M) and a collected dataset (22M), all annotated using InternVL2.
- Synthetic Data: Over 60M images generated via the collage pipeline, covering T2I, instruction-based editing, inpainting, outpainting, drag editing, and reference image generation.
- Segmentation/Detection: An additional 8M synthetic images were created to help the model learn spatial grounding.
Key Insights and Ideas
- Synergistic Training: The authors found that T2I training helps the model understand concepts, while editing training helps it grasp the logic of instructions. Training on both simultaneously prevents "concept forgetting" and improves overall generation quality.
- VLM as a Unified Encoder: Using a VLM instead of a simple text encoder is a game-changer. It allows the model to "see" a reference image (for subject-driven tasks) and "read" an instruction (for editing tasks) without needing separate adapters or encoders.
- Efficiency of Scale: The study reveals that DiT's superiority over UNet stems from its computational allocation. By focusing on the $2\times$ downsampled latent space, DreamOmni converges up to four times faster than models like SD3-Medium.
Key Results
DreamOmni demonstrates impressive versatility across several benchmarks:
- T2I Generation: On the GenEval benchmark, DreamOmni achieved an overall score of 0.70, matching the state-of-the-art SD3-Medium while outperforming it in specific categories like object positioning and color attribution.
- Inpainting: Quantitative tests showed DreamOmni significantly outperforms ControlNet-Inpainting and SD-Inpainting, particularly in maintaining coherence over large masked areas.
- Instruction & Drag Editing: Qualitative results show that DreamOmni follows complex instructions (e.g., "Replace the owl with a hawk") with much higher fidelity and fewer artifacts than previous methods like MGIE or InstructP2P.
Limitations and Future Work
Despite its strengths, the authors note that large-scale rotation in drag-based editing remains a challenge. Because rotation involves complex 3D-like transformations of an object's internal structure, the model can sometimes produce deformations.
Future work may involve further scaling the model parameters and refining the synthetic pipeline to include more complex 3D transformations and lighting consistency to make the "collages" even more realistic.
Summary updated