UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts

Fu-Yun Wang, Han Zhang, Michaël Gharbi, Hongsheng Li, Taesung ParkarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) and Diffusion Models (DMs) have revolutionized text and image generation respectively, they are typically treated as isolated systems. Emerging "unified" models attempt to bridge this gap, but training them effectively remains a significant challenge. Language models operate on discrete tokens, while diffusion models function through continuous noise-clearing steps. This fundamental difference in data representation makes joint optimization difficult, often resulting in models that struggle with complex tasks like nuanced image editing or self-correction.

UniRL-Zero is a novel reinforcement learning (RL) framework designed to address this by jointly optimizing multimodal language models and diffusion models within a single, unified architecture. By enabling these "cooperative experts" to learn together, UniRL-Zero allows a model to "think" before it "paints" and "reflect" after it "creates."

The UniRL-Zero Framework

The UniRL-Zero architecture utilizes a frozen Qwen2.5-VL as the language backbone and a SANA-1.6B linear transformer as the diffusion expert. These components are linked via meta-query tokens and a bidirectional connector, allowing for seamless information flow between the text and image domains.

The UniRL-Zero framework defines six interaction scenarios and a joint optimization workflow using multimodal language models and diffusion experts.

To optimize this hybrid system, the authors adapted Group Relative Policy Optimization (GRPO). This efficient RL algorithm eliminates the need for a separate "critic" model and is modified to handle both discrete token losses (for the LM) and continuous denoising losses (for the DM). The framework supports six distinct interaction scenarios, including reasoning-enhanced text-to-image generation, instructional image editing, and a self-reflective loop where the model identifies and corrects its own visual errors.

Methodology and Training

UniRL-Zero introduces several innovative training techniques:

  • Cycle Edit RL: Inspired by CycleGAN, this method uses a cycle-consistency reward for image editing. The model edits an image and then attempts to reverse the process; a high reward is granted if the reversed image matches the original.
  • Reflective Loop: The LM is trained to inspect generated images, identify discrepancies (e.g., incorrect object counts), and trigger refinement cycles.
  • Synthetic Reasoning Data: To "cold start" the model's reasoning, the researchers used Claude to generate 10,000 high-quality Chain-of-Thought (CoT) prompts and reflection-augmented data points.

Key Insights and Results

The most significant finding is that RL directly encourages the LM to generate longer, more descriptive reasoning chains before passing instructions to the diffusion expert. This "thinking time" significantly improves compositional accuracy in the final image.

Training curves showing the increase in GenEval rewards and the evolution of reasoning lengths throughout the RL process.

Empirical results demonstrate the power of this joint optimization:

  • Compositional Accuracy: The base 1.6B model achieved a 0.69 GenEval score, which jumped to 0.85 after CoT-enhanced RL. This outperforms much larger models like SDXL (0.55) and PixArt-Σ (0.52).
  • Self-Correction: RL training improved the model's ability to identify its own errors ("Judge Accuracy") from 72% to over 81%, and its ability to fix those errors ("Correction Accuracy") from 48% to 74%.

Conclusion and Future Work

UniRL-Zero proves that backpropagating rewards through a complex pipeline of discrete text and continuous image generation is not only possible but highly effective. While the current framework relies on CLIP-based rewards and faces some constraints regarding experimental scale, it sets a new benchmark for unified multimodal models. Future work will likely focus on scaling the diffusion expert and exploring ways to maintain the language model's full plasticity during the RL process without degrading its general reasoning capabilities.

Summary updated

Abstract

We present UniRL-Zero, a unified reinforcement learning (RL) framework that boosts, multimodal language model understanding and reasoning, diffusion model multimedia generation, and their beneficial interaction capabilities within a unified model. Our work defines six scenarios for unified model reinforcement learning, providing systematic baselines for reinforcement learning of unified understanding and generation model. Our code is available at https://github.com/G-U-N/UniRL.
BibTeX
@misc{wang2025unirl,
  author = {Fu-Yun Wang and Han Zhang and Michaël Gharbi and Hongsheng Li and Taesung Park},
  title = {{UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts}},
  year = {2025},
  eprint = {2510.17937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.17937},
}
Figure