TL;DR
UniFusion introduces a framework that uses a single, frozen Large Vision-Language Model (VLM) as a unified encoder for both text and images in diffusion-based image generation. By leveraging intermediate VLM layers and an in-model prompt rewriting mechanism, UniFusion achieves state-of-the-art text-to-image generation and complex image editing without requiring resource-intensive joint multimodal training.
What Problem Does This Paper Address?
Most state-of-the-art image generation models rely on separate, heterogeneous encoding spaces: text encoders (like T5) capture high-level semantics, while image encoders (like VAEs or CLIP) preserve low-level pixel details. This architectural separation forces the downstream Diffusion Transformer (DiT) to expend significant capacity simply aligning these disparate feature spaces. This mismatch becomes highly apparent in image editing tasks, where models frequently struggle to balance instruction adherence with content preservation.
While natively multimodal models can bridge this gap, training them from scratch requires massive datasets and substantial computational power. UniFusion addresses this by utilizing a pre-trained, frozen VLM as a single, unified semantic space for both text and image conditioning.
Methods
The UniFusion architecture consists of an 8B VLM (InternVL3-8B) acting as the unified encoder paired with an 8B Diffusion Transformer (DiT). The framework introduces two core components to effectively bridge the VLM and the generative model:
- Layerwise Attention Pooling (LAP): Prior attempts to use VLMs typically extract features only from the final hidden layer, which tends to lose fine-grained visual details. LAP extracts features from multiple intermediate layers (specifically every third layer). It uses learnable self-attention blocks to aggregate these features into a single dense representation, capturing both low-level details and high-level abstractions.
- Bi-directional Refiner: Auto-regressive VLMs suffer from position bias due to causal attention masking (tokens mentioned late are insufficiently represented). UniFusion adds a small bi-directional transformer refiner after LAP to mitigate this bias.
- VLM-Enabled Rewriting Injection with Flexible Inference (Verifi): Instead of using a standalone prompt rewriter, Verifi instructs the frozen VLM to rewrite the user prompt in-context during a single forward pass. The DiT is then conditioned on these rewritten tokens, directly embedding the VLM's world knowledge and reasoning capabilities into the generation process.
Datasets
The model was trained on a mixture of text-to-image, image reconstruction, and image-text pairs totaling fewer than 1 billion samples, followed by instruction-based image editing data (~10k steps).
For evaluation, the authors utilized:
- GenAI-Bench and GenEval for text-to-image prompt adherence.
- DPG-Bench: The authors identified a critical score aggregation bug in the official DPG-Bench repository that had caused prior papers to report mathematically impossible results. They re-implemented the benchmark evaluation script and upgraded the underlying VQA evaluator to Gemma-3-27b-it with extensive Chain-of-Thought reasoning to ensure reliable, unbiased scoring.
Key Insights and Ideas
- Middle Layers Hold the Key: By visualizing the learnable weights within the LAP module, the authors discovered that the model naturally prioritizes early-to-middle VLM layers. Dropping the first or last layers has minimal impact on the final image, whereas removing middle layers severely degrades scene composition.
- Cross-Task Positive Transfer: Continual training on image editing and reference tasks doesn't just add new capabilities—it actively improves the model's baseline text-to-image prompt adherence and aesthetic quality.
- Emergent Zero-Shot Generalization: Despite being trained exclusively on single-reference image editing data, UniFusion naturally generalizes to multi-reference image composition, handling mixed aspect ratios, resolutions, and perspective shifts entirely zero-shot.
Key Results
- Benchmark Performance: On the refined DPG-Bench, UniFusion (8B DiT) achieved a Macro Average score of 0.731, outperforming much larger models like Bagel (14B MoT) at 0.715 and Flux.1 [dev] (12B) at 0.693, despite using a significantly smaller training set.
- Efficiency: The authors demonstrated that instead of training from scratch, a model can be successfully adapted from an early T5-conditioned checkpoint to the VLM-conditioned LAP setup in just ~10k steps with zero performance loss.
- Qualitative Excellence: UniFusion shows strong instruction-following capabilities on complex, long-form prompts and produces photorealistic outputs with fewer over-saturation or smoothing artifacts common in contemporary models.
Limitations and Future Work
- Computational Overhead: Performing auto-regressive prompt rewriting with an 8B transformer during the encoding phase increases inference latency and compute requirements.
- Text Rendering Vulnerabilities: While the diffusion model is capable of generating typography, it inherits spelling mistakes directly from the underlying VLM (InternVL), which occasionally hallucinates typos in the rewritten prompts. Future work includes exploring alternative VLM backbones to alleviate this bottleneck.
Summary updated