DreamOmni: Unified Image Generation and Editing

Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia

TL;DR

DreamOmni is a unified generative model that seamlessly integrates text-to-image (T2I) generation with a wide array of image editing tasks—such as drag-based editing, inpainting, and subject-driven generation—within a single framework. By leveraging a novel synthetic collage data pipeline and a Vision-Language Model (VLM) backbone, it achieves state-of-the-art performance while significantly simplifying the deployment of multi-task vision systems.

What Problem Does This Paper Address?

While text-to-image models have reached incredible heights, they remain fragmented. If you want to edit an image, you typically need specialized "plugins" like ControlNet, IP-Adapter, or models with modified input channels like InstructP2P. This reliance on specialized frameworks makes multi-tasking difficult and deployment cumbersome.

Furthermore, training these models requires high-quality "paired" data (e.g., an image before and after a specific edit). Such data is notoriously difficult to collect or synthesize accurately; manual labeling is slow, and automated methods often introduce artifacts or fail to follow complex instructions. DreamOmni seeks to bridge this gap by creating a native, unified model trained on a massive, programmatically generated synthetic dataset.

Methods

The authors propose a unified architecture and a scalable data generation strategy:

  1. The Framework: DreamOmni uses a 2.5B parameter Diffusion Transformer (DiT) as its backbone. Unlike traditional models that use CLIP or T5 for text, DreamOmni adopts a Vision-Language Model (VLM)—specifically Qwen2-VL—as its encoder. This allows the model to process text instructions and reference images in a unified latent space.
  2. Architectural Refinements: To improve training speed, the authors incorporated UNet-style residual connections into the DiT blocks. They discovered that concentrating computations on $2\times$ downsampled latents (rather than spreading them across multiple scales) provides the best trade-off between performance and efficiency.
  3. Synthetic Collage Pipeline: To solve the data scarcity problem, the researchers developed a pipeline that uses "sticker-like" elements. By programmatically placing, scaling, rotating, and replacing these elements on a canvas, they can generate billions of perfectly paired training examples for tasks like "add a hat" or "move the cat to the left."
The DreamOmni framework (a) and examples of the synthetic collage data pipeline (b).

Datasets

DreamOmni was trained on a massive scale:

  • Real-world Data: 125M images, including the LAION dataset (103M) and a collected dataset (22M), all annotated using InternVL2.
  • Synthetic Data: Over 60M images generated via the collage pipeline, covering T2I, instruction-based editing, inpainting, outpainting, drag editing, and reference image generation.
  • Segmentation/Detection: An additional 8M synthetic images were created to help the model learn spatial grounding.

Key Insights and Ideas

  • Synergistic Training: The authors found that T2I training helps the model understand concepts, while editing training helps it grasp the logic of instructions. Training on both simultaneously prevents "concept forgetting" and improves overall generation quality.
  • VLM as a Unified Encoder: Using a VLM instead of a simple text encoder is a game-changer. It allows the model to "see" a reference image (for subject-driven tasks) and "read" an instruction (for editing tasks) without needing separate adapters or encoders.
  • Efficiency of Scale: The study reveals that DiT's superiority over UNet stems from its computational allocation. By focusing on the $2\times$ downsampled latent space, DreamOmni converges up to four times faster than models like SD3-Medium.
A comparison of different frameworks showing DreamOmni's superior convergence speed and FID scores.

Key Results

DreamOmni demonstrates impressive versatility across several benchmarks:

  • T2I Generation: On the GenEval benchmark, DreamOmni achieved an overall score of 0.70, matching the state-of-the-art SD3-Medium while outperforming it in specific categories like object positioning and color attribution.
  • Inpainting: Quantitative tests showed DreamOmni significantly outperforms ControlNet-Inpainting and SD-Inpainting, particularly in maintaining coherence over large masked areas.
  • Instruction & Drag Editing: Qualitative results show that DreamOmni follows complex instructions (e.g., "Replace the owl with a hawk") with much higher fidelity and fewer artifacts than previous methods like MGIE or InstructP2P.
Qualitative comparison of T2I generation across different models.

Limitations and Future Work

Despite its strengths, the authors note that large-scale rotation in drag-based editing remains a challenge. Because rotation involves complex 3D-like transformations of an object's internal structure, the model can sometimes produce deformations.

Future work may involve further scaling the model parameters and refining the synthetic pipeline to include more complex 3D transformations and lighting consistency to make the "collages" even more realistic.

Summary updated

Abstract

Currently, the success of large language models (LLMs) illustrates that a unified multitasking approach can significantly enhance model usability, streamline deployment, and foster synergistic benefits across different tasks. However, in computer vision, while text-to-image (T2I) models have significantly improved generation quality through scaling up, their framework design did not initially consider how to unify with downstream tasks, such as various types of editing. To address this, we introduce DreamOmni, a unified model for image generation and editing. We begin by analyzing existing frameworks and the requirements of downstream tasks, proposing a unified framework that integrates both T2I models and various editing tasks. Furthermore, another key challenge is the efficient creation of high-quality editing data, particularly for instruction-based and drag-based editing. To this end, we develop a synthetic data pipeline using sticker-like elements to synthesize accurate, high-quality datasets efficiently, which enables editing data scaling up for unified model training. For training, DreamOmni jointly trains T2I generation and downstream tasks. T2I training enhances the model's understanding of specific concepts and improves generation quality, while editing training helps the model grasp the nuances of the editing task. This collaboration significantly boosts editing performance. Extensive experiments confirm the effectiveness of DreamOmni. The code and model will be released.
BibTeX
@inproceedings{xia2025dreamomni,
  author = {Bin Xia and Yuechen Zhang and Jingyao Li and Chengyao Wang and Yitong Wang and Xinglong Wu and Bei Yu and Jiaya Jia},
  title = {{DreamOmni: Unified Image Generation and Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02657},
}
Figure