DeCoRL: Decomposed Consistency Reinforcement Learning for Multi-Image Composition

Zhiqiang Wu, Shuang Sun, Jiale Zhang, Jing Li, Yibo Chen, Xu Tang, Yao HuarXiv 2026 · arXiv

TL;DR

DeCoRL introduces a reinforcement learning framework for multi-image composition that solves the "consistency collapse" seen when models try to merge many reference images at once. By decomposing a complex holistic scene into individual object-level consistency checks, the model learns to preserve identities and attributes far more reliably than standard supervised fine-tuning.

What Problem Does This Paper Address?

Multi-image composition (MICo) is the task of taking several distinct reference images—say, a specific woman, a pair of pants, and a particular vase—and blending them into a single, coherent scene based on a text prompt.

While modern diffusion models are great at single-image editing, they struggle as the number of references grows. The "many-to-one" correspondence problem leads to identity confusion (the woman looks different), attribute leakage (the pants' color bleeds onto the shirt), or simply missing objects. Standard Supervised Fine-Tuning (SFT) provides only implicit supervision on the final image, failing to teach the model explicit object-level correspondence. Furthermore, using a large vision-language model (VLM) to score the entire composed image is often unreliable because the VLM itself struggles to attend to multiple fine-grained references simultaneously.

Methods

The authors propose DeCoRL (Decomposed Consistency Reinforcement Learning), which shifts the focus from global scene evaluation to local object verification.

The DeCoRL framework decomposes holistic consistency into object-level rewards to guide RL post-training.

The framework consists of three main stages:

  1. Instruction-Relevant Object Correspondence: A VLM parses the text prompt to identify which objects from which reference images are required. Then, Grounding DINO localizes these objects in both the reference images and the newly generated image.
  2. Single-Object Consistency Reward Model: The authors fine-tuned a Qwen3-VL-32B model to act as a specialized judge. Crucially, it was trained using Chain-of-Thought (CoT) reasoning to prevent it from taking "shortcuts." Instead of a simple yes/no, it explains its reasoning. The reward signal is derived from the logit probabilities of the "yes" and "no" tokens, providing a continuous gradient for optimization.
  3. Multi-Object Reward Aggregation & RL: The individual object scores are averaged into a global reward. If a required object is missing from the generated image, it receives a score of zero. This aggregate reward is then used to post-train the diffusion model using Diffusion NFT, an efficient RL fine-tuning method.

Datasets

  • Training: The model was trained on a combination of MICo-150K, UniPic 3.0, and Nano-consistent-150K (for single-image consistency).
  • MICo-Bench: An existing public benchmark for multi-image composition.
  • DeCo-Bench (New): Introduced in this paper, this benchmark specifically targets the scaling challenge, featuring test cases with 2 to 6 reference images. It uses real-world photographs and evaluates consistency using a high-strength evaluator (Gemini-3.1-Pro).

Key Insights and Ideas

  • Decomposition is Tractable: The core "aha" moment is that while judging a whole scene is "ill-posed" and difficult for current VLMs, judging if "Object A in Image 1 matches Object A in the Result" is a much simpler, more reliable task.
  • CoT for Rewards: Training the reward model with Chain-of-Thought significantly improves its accuracy. Without it, reward models often overfit to low-entropy "yes/no" signals and fail to provide meaningful feedback to the RL agent.
  • Continuous Logit Rewards: By using the difference between "yes" and "no" token logits rather than a hard binary score, the framework provides a smoother optimization landscape for reinforcement learning.
  • Addressing the "Collapse": The authors observed that baseline models often "collapse" when dealing with 5-6 reference images. DeCoRL specifically mitigates this by explicitly penalizing missing objects in the reward function.

Key Results

DeCoRL achieves state-of-the-art performance across multiple benchmarks, particularly excelling as the complexity of the scene increases.

  • DeCo-Bench: DeCoRL achieved an overall score of 8.16, significantly outperforming the SFT baseline (7.05) and open-source models like FLUX.2-dev (7.10). It even surpassed the proprietary Seedream 4.0 (7.76).
  • MICo-Bench: The model reached a state-of-the-art overall score of 54.03, showing massive gains in the "Human Centric" and "De&Re" (Decomposition and Recomposition) categories compared to the baseline.
  • Qualitative Improvement: As shown in Figure 1, DeCoRL maintains the specific patterns of clothing and the identities of objects (like a specific camera or chair) where other models produce generic or inconsistent versions.
Qualitative comparisons show that DeCoRL preserves object identities and attributes much better than open-source and proprietary baselines.

Limitations and Future Work

  • Local vs. Global: The current reward focuses on localizable objects. It does not yet explicitly handle abstract styles, complex spatial relations, or non-localizable consistency (like overall lighting mood).
  • Grounding Dependency: The system's performance is capped by the accuracy of the grounding model (Grounding DINO). If the detector fails to find an object, the reward signal becomes noisy.
  • Computational Cost: The pipeline involves multiple large models (VLM for parsing, Grounding DINO, VLM for reward), which increases the resource requirements for training.
  • Scaling Beyond Six: The current study stops at six reference images; future work is needed to see if these principles hold as the reference count continues to scale.

Summary updated

Abstract

Multi-image composition (MICo) is a critical yet challenging capability for large-scale image editing models, where consistency degrades rapidly as the number of reference images increases. While supervised fine-tuning (SFT) can equip models with basic MICo ability, it provides only implicit supervision on the composed image and often fails to learn reliable object-level correspondence across multiple references. We propose DeCoRL, a reinforcement-learning (RL) framework that improves MICo consistency by decomposing holistic MICo consistency into object-level consistency. DeCoRL first establishes instruction-relevant object correspondences across reference images and localizes the corresponding regions in the generated image. It then uses a VLM-based reward model trained on single-object consistency annotations to produce fine-grained, object-wise feedback, which is aggregated into a multi-object consistency reward to guide RL post-training. We further introduce DeCo-Bench, a benchmark covering compositions from 2 to 6 reference images. Experiments on DeCo-Bench and the public MICo-Bench demonstrate consistent improvements in MICo consistency.
BibTeX
@misc{wu2026decorl,
  author = {Zhiqiang Wu and Shuang Sun and Jiale Zhang and Jing Li and Yibo Chen and Xu Tang and Yao Hu},
  title = {{DeCoRL: Decomposed Consistency Reinforcement Learning for Multi-Image Composition}},
  year = {2026},
  eprint = {2610.39265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39265},
}
Figure