TL;DR
DeCoRL introduces a reinforcement learning framework for multi-image composition that solves the "consistency collapse" seen when models try to merge many reference images at once. By decomposing a complex holistic scene into individual object-level consistency checks, the model learns to preserve identities and attributes far more reliably than standard supervised fine-tuning.
What Problem Does This Paper Address?
Multi-image composition (MICo) is the task of taking several distinct reference images—say, a specific woman, a pair of pants, and a particular vase—and blending them into a single, coherent scene based on a text prompt.
While modern diffusion models are great at single-image editing, they struggle as the number of references grows. The "many-to-one" correspondence problem leads to identity confusion (the woman looks different), attribute leakage (the pants' color bleeds onto the shirt), or simply missing objects. Standard Supervised Fine-Tuning (SFT) provides only implicit supervision on the final image, failing to teach the model explicit object-level correspondence. Furthermore, using a large vision-language model (VLM) to score the entire composed image is often unreliable because the VLM itself struggles to attend to multiple fine-grained references simultaneously.
Methods
The authors propose DeCoRL (Decomposed Consistency Reinforcement Learning), which shifts the focus from global scene evaluation to local object verification.
The framework consists of three main stages:
- Instruction-Relevant Object Correspondence: A VLM parses the text prompt to identify which objects from which reference images are required. Then, Grounding DINO localizes these objects in both the reference images and the newly generated image.
- Single-Object Consistency Reward Model: The authors fine-tuned a Qwen3-VL-32B model to act as a specialized judge. Crucially, it was trained using Chain-of-Thought (CoT) reasoning to prevent it from taking "shortcuts." Instead of a simple yes/no, it explains its reasoning. The reward signal is derived from the logit probabilities of the "yes" and "no" tokens, providing a continuous gradient for optimization.
- Multi-Object Reward Aggregation & RL: The individual object scores are averaged into a global reward. If a required object is missing from the generated image, it receives a score of zero. This aggregate reward is then used to post-train the diffusion model using Diffusion NFT, an efficient RL fine-tuning method.
Datasets
- Training: The model was trained on a combination of MICo-150K, UniPic 3.0, and Nano-consistent-150K (for single-image consistency).
- MICo-Bench: An existing public benchmark for multi-image composition.
- DeCo-Bench (New): Introduced in this paper, this benchmark specifically targets the scaling challenge, featuring test cases with 2 to 6 reference images. It uses real-world photographs and evaluates consistency using a high-strength evaluator (Gemini-3.1-Pro).
Key Insights and Ideas
- Decomposition is Tractable: The core "aha" moment is that while judging a whole scene is "ill-posed" and difficult for current VLMs, judging if "Object A in Image 1 matches Object A in the Result" is a much simpler, more reliable task.
- CoT for Rewards: Training the reward model with Chain-of-Thought significantly improves its accuracy. Without it, reward models often overfit to low-entropy "yes/no" signals and fail to provide meaningful feedback to the RL agent.
- Continuous Logit Rewards: By using the difference between "yes" and "no" token logits rather than a hard binary score, the framework provides a smoother optimization landscape for reinforcement learning.
- Addressing the "Collapse": The authors observed that baseline models often "collapse" when dealing with 5-6 reference images. DeCoRL specifically mitigates this by explicitly penalizing missing objects in the reward function.
Key Results
DeCoRL achieves state-of-the-art performance across multiple benchmarks, particularly excelling as the complexity of the scene increases.
- DeCo-Bench: DeCoRL achieved an overall score of 8.16, significantly outperforming the SFT baseline (7.05) and open-source models like FLUX.2-dev (7.10). It even surpassed the proprietary Seedream 4.0 (7.76).
- MICo-Bench: The model reached a state-of-the-art overall score of 54.03, showing massive gains in the "Human Centric" and "De&Re" (Decomposition and Recomposition) categories compared to the baseline.
- Qualitative Improvement: As shown in Figure 1, DeCoRL maintains the specific patterns of clothing and the identities of objects (like a specific camera or chair) where other models produce generic or inconsistent versions.
Limitations and Future Work
- Local vs. Global: The current reward focuses on localizable objects. It does not yet explicitly handle abstract styles, complex spatial relations, or non-localizable consistency (like overall lighting mood).
- Grounding Dependency: The system's performance is capped by the accuracy of the grounding model (Grounding DINO). If the detector fails to find an object, the reward signal becomes noisy.
- Computational Cost: The pipeline involves multiple large models (VLM for parsing, Grounding DINO, VLM for reward), which increases the resource requirements for training.
- Scaling Beyond Six: The current study stops at six reference images; future work is needed to see if these principles hold as the reference count continues to scale.
Summary updated