TL;DR
MORE (Merge Once, Route Early) is a training-free pipeline that enables high-fidelity composition of multiple personalized subjects from a large pool of LoRAs (up to $K=20$) within the FLUX.1 diffusion model. By combining a robust offline weight-merging strategy with a novel spatio-temporal routing mechanism during inference, it achieves a subject coexistence rate of over 94%, far outperforming existing state-of-the-art methods.
What Problem Does This Paper Address?
Personalizing text-to-image models using Low-Rank Adaptation (LoRA) has become a standard practice. However, trying to render multiple personalized subjects (e.g., "my cat" and "my favorite robot toy") in a single image remains notoriously difficult.
On large models like FLUX, two main failure modes emerge:
- Pool Crosstalk: When multiple LoRAs are loaded, their weights interfere. Since many community LoRAs use the same trigger word (like
sks), the model gets confused, leading to subjects vanishing or fusing into weird hybrids. - Coexistence Collapse: Even if you only load the two required LoRAs, the model's joint attention often fails to bind identities to specific locations. This results in "identity mixing" or the image splitting into a disjointed collage.
Existing solutions either try to merge weights statically (which lacks spatial control) or use spatial masking (which often fails to prevent identity leakage).
Methods
The authors propose a two-stage pipeline:
Stage 1: Merge Once (Offline)
Instead of loading LoRAs on the fly, MORE merges the entire pool of $K$ LoRAs into a single, fixed-rank adapter.
- Subspace-Regression Merge: It uses a regularized version of SSR-Merge, optimized with a "relative ridge" to handle layers with different feature scales.
- Calibration: To prevent the merge from being dominated by padding tokens, they calibrate using noise latents and diverse "scene" prompts.
- Exact Factorization: The merged weight is mathematically factored into a shared part (common features) and per-subject parts. This allows for precise routing later.
Stage 2: Route Early (Inference)
During the image generation process, MORE applies three key techniques to ensure subjects stay in their lanes:
- Early Layout Gating: For the first 30% of denoising steps—where the model decides layout and identity—subject-specific weights are "gated" to their respective regions. After 30%, the full static merge takes over to render fine details.
- Soft Text-Region Attention Bias: A decaying cosine bias is added to the attention logits, gently encouraging image regions to attend more to their corresponding subject tokens in the prompt.
- Cyclic Swap Guidance: This is the "secret sauce." The model performs an extra pass where subjects are intentionally bound to the wrong regions. By extrapolating away from this "bad" prediction (similar to Classifier-Free Guidance), the model is forced to sharpen the distinctions between subjects and prevent them from looking alike.
Datasets
- DreamBooth Subjects: 20 diverse subjects (animals, toys, objects) trained as rank-32 LoRAs on FLUX.1-dev.
- Pair Benchmark: A newly introduced evaluation set consisting of 20 prompts across multiple seeds, testing combinations of subjects in different scenes (e.g., "on a wooden table," "in a grassy field").
- Three-Subject Test: Evaluation on prompts requiring three distinct personalized subjects.
Key Insights and Ideas
- The "Early" Window: The authors found that identity and layout are committed in the first ~30% of denoising steps. Routing throughout the entire process is unnecessary and can actually hurt image quality.
- Soft vs. Hard Masks: Unlike previous methods that use hard $- \infty$ masks to block attention, MORE uses a soft, decaying bias. This allows the model to maintain global coherence while still enforcing local identity.
- Guidance as a Contrastive Tool: The Cyclic Swap Guidance is a clever application of guidance. By showing the model what a wrong binding looks like, it learns to better isolate the unique features of each subject.
Key Results
MORE significantly outperforms baselines like FreeFuse, LoRAShop, and Regional Prompting:
- Coexistence (BOTHC): For a pool of $K=10$, MORE keeps both subjects in 94.2% of images, compared to 50.8% for FreeFuse and 25.0% for LoRAShop.
- Large Pools: Even with $K=20$ subjects merged into one adapter, it maintains a 78.3% success rate.
- Human Preference: In a blind A/B study, MORE was preferred over all baselines in nearly 100% of trials regarding identity retention and text alignment.
- Efficiency: Despite the extra guidance pass, it generates images in ~6.9 seconds, which is faster than several mask-inference baselines.
Limitations and Future Work
- Layout Dependency: MORE requires a layout (bounding boxes). While these can be inferred automatically from the base model's attention, doing so drops the success rate by about 24%.
- Small Regions: The method struggles when a subject is assigned to a very small region, as it doesn't collect enough "attention mass" to establish identity.
- Color Saturation: The attention bias can slightly increase color saturation, though the authors mitigate this with a decaying schedule.
- Future Work: Improving automatic layout inference and extending the method to handle even more complex compositions (e.g., many small subjects) are logical next steps.
Summary updated