UniFusion: Vision-Language Model as Unified Encoder in Image Generation

Kevin Li, Manuel Brack, Sudeep Katakol, Hareesh Ravi, Ajinkya Kale

TL;DR

UniFusion introduces a framework that uses a single, frozen Large Vision-Language Model (VLM) as a unified encoder for both text and images in diffusion-based image generation. By leveraging intermediate VLM layers and an in-model prompt rewriting mechanism, UniFusion achieves state-of-the-art text-to-image generation and complex image editing without requiring resource-intensive joint multimodal training.

What Problem Does This Paper Address?

Most state-of-the-art image generation models rely on separate, heterogeneous encoding spaces: text encoders (like T5) capture high-level semantics, while image encoders (like VAEs or CLIP) preserve low-level pixel details. This architectural separation forces the downstream Diffusion Transformer (DiT) to expend significant capacity simply aligning these disparate feature spaces. This mismatch becomes highly apparent in image editing tasks, where models frequently struggle to balance instruction adherence with content preservation.

While natively multimodal models can bridge this gap, training them from scratch requires massive datasets and substantial computational power. UniFusion addresses this by utilizing a pre-trained, frozen VLM as a single, unified semantic space for both text and image conditioning.

Methods

The UniFusion architecture consists of an 8B VLM (InternVL3-8B) acting as the unified encoder paired with an 8B Diffusion Transformer (DiT). The framework introduces two core components to effectively bridge the VLM and the generative model:

  1. Layerwise Attention Pooling (LAP): Prior attempts to use VLMs typically extract features only from the final hidden layer, which tends to lose fine-grained visual details. LAP extracts features from multiple intermediate layers (specifically every third layer). It uses learnable self-attention blocks to aggregate these features into a single dense representation, capturing both low-level details and high-level abstractions.
  2. Bi-directional Refiner: Auto-regressive VLMs suffer from position bias due to causal attention masking (tokens mentioned late are insufficiently represented). UniFusion adds a small bi-directional transformer refiner after LAP to mitigate this bias.
  3. VLM-Enabled Rewriting Injection with Flexible Inference (Verifi): Instead of using a standalone prompt rewriter, Verifi instructs the frozen VLM to rewrite the user prompt in-context during a single forward pass. The DiT is then conditioned on these rewritten tokens, directly embedding the VLM's world knowledge and reasoning capabilities into the generation process.
Overview of the UniFusion architecture and inference paradigm.

Datasets

The model was trained on a mixture of text-to-image, image reconstruction, and image-text pairs totaling fewer than 1 billion samples, followed by instruction-based image editing data (~10k steps).

For evaluation, the authors utilized:

  • GenAI-Bench and GenEval for text-to-image prompt adherence.
  • DPG-Bench: The authors identified a critical score aggregation bug in the official DPG-Bench repository that had caused prior papers to report mathematically impossible results. They re-implemented the benchmark evaluation script and upgraded the underlying VQA evaluator to Gemma-3-27b-it with extensive Chain-of-Thought reasoning to ensure reliable, unbiased scoring.

Key Insights and Ideas

  • Middle Layers Hold the Key: By visualizing the learnable weights within the LAP module, the authors discovered that the model naturally prioritizes early-to-middle VLM layers. Dropping the first or last layers has minimal impact on the final image, whereas removing middle layers severely degrades scene composition.
  • Cross-Task Positive Transfer: Continual training on image editing and reference tasks doesn't just add new capabilities—it actively improves the model's baseline text-to-image prompt adherence and aesthetic quality.
  • Emergent Zero-Shot Generalization: Despite being trained exclusively on single-reference image editing data, UniFusion naturally generalizes to multi-reference image composition, handling mixed aspect ratios, resolutions, and perspective shifts entirely zero-shot.
Zero-shot multi-reference and image editing capabilities.

Key Results

  • Benchmark Performance: On the refined DPG-Bench, UniFusion (8B DiT) achieved a Macro Average score of 0.731, outperforming much larger models like Bagel (14B MoT) at 0.715 and Flux.1 [dev] (12B) at 0.693, despite using a significantly smaller training set.
  • Efficiency: The authors demonstrated that instead of training from scratch, a model can be successfully adapted from an early T5-conditioned checkpoint to the VLM-conditioned LAP setup in just ~10k steps with zero performance loss.
  • Qualitative Excellence: UniFusion shows strong instruction-following capabilities on complex, long-form prompts and produces photorealistic outputs with fewer over-saturation or smoothing artifacts common in contemporary models.
Diverse textual image editing and image reference workflows.

Limitations and Future Work

  • Computational Overhead: Performing auto-regressive prompt rewriting with an 8B transformer during the encoding phase increases inference latency and compute requirements.
  • Text Rendering Vulnerabilities: While the diffusion model is capable of generating typography, it inherits spelling mistakes directly from the underlying VLM (InternVL), which occasionally hallucinates typos in the rewritten prompts. Future work includes exploring alternative VLM backbones to alleviate this bottleneck.

Summary updated

Abstract

Although recent advances in visual generation have been remarkable, most existing architectures still depend on distinct encoders for images and text. This separation constrains diffusion models' ability to perform cross-modal reasoning and knowledge transfer. Prior attempts to bridge this gap often use the last layer information from VLM, employ multiple visual encoders, or train large unified models jointly for text and image generation, which demands substantial computational resources and large-scale data, limiting its accessibility.We present UniFusion, a diffusion-based generative model conditioned on a frozen large vision-language model (VLM) that serves as a unified multimodal encoder. At the core of UniFusion is the Layerwise Attention Pooling (LAP) mechanism that extracts both high level semantics and low level details from text and visual tokens of a frozen VLM to condition a diffusion generative model. We demonstrate that LAP outperforms other shallow fusion architectures on text-image alignment for generation and faithful transfer of visual information from VLM to the diffusion model which is key for editing. We propose VLM-Enabled Rewriting Injection with Flexibile Inference (VERIFI), which conditions a diffusion transformer (DiT) only on the text tokens generated by the VLM during in-model prompt rewriting. VERIFI combines the alignment of the conditioning distribution with the VLM's reasoning capabilities for increased capabilities and flexibility at inference. In addition, finetuning on editing task not only improves text-image alignment for generation, indicative of cross-modality knowledge transfer, but also exhibits tremendous generalization capabilities. Our model when trained on single image editing, zero-shot generalizes to multiple image references further motivating the unified encoder design of UniFusion.
BibTeX
@misc{li2025unifusion,
  author = {Kevin Li and Manuel Brack and Sudeep Katakol and Hareesh Ravi and Ajinkya Kale},
  title = {{UniFusion: Vision-Language Model as Unified Encoder in Image Generation}},
  year = {2025},
  eprint = {2510.12789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12789},
}
Figure