TL;DR
CARE-Edit introduces a "Condition-Aware Routing of Experts" framework that solves task interference in unified image editors. By dynamically dispatching image tokens to specialized experts (Text, Mask, Reference, and Base) based on the current diffusion timestep and input signals, it achieves high-fidelity results in complex tasks like subject replacement and style transfer without the artifacts common in static models.
What Problem Does This Paper Address?
Current unified diffusion-based image editors (like ControlNet or OmniControl) often struggle when multiple conditioning signals—such as a text prompt, a spatial mask, and a reference image—are provided simultaneously.
The authors identify a core issue: static fusion. Most models simply concatenate these signals or use additive adapters, forcing a single shared backbone to process everything at once. This leads to "task interference," where signals conflict. For example, a text prompt might override a mask boundary (causing color bleeding), or a reference image might cause "identity drift" where the subject doesn't quite match the original. Furthermore, the importance of these signals changes during the generation process (e.g., layout matters early on, while fine details matter later), but static models can't adapt to this temporal shift.
Methods
The researchers propose CARE-Edit, which moves away from a "one-size-fits-all" backbone toward a Mixture-of-Experts (MoE) architecture. Key components include:
- Heterogeneous Experts: The model employs four specialized, lightweight adapters (experts):
- Text Expert: Handles semantic reasoning and object synthesis.
- Mask Expert: Focuses on spatial precision and boundary refinement.
- Reference Expert: Manages identity and style-consistent transformations.
- Base Expert: Ensures global coherence and background preservation.
- Latent-Attention Router: A lightweight module that uses sparse top-K selection to dynamically assign each diffusion token to the most relevant experts. This routing is "condition-aware," meaning it looks at the prompt, mask, and reference features to decide which expert should take the lead for a specific part of the image.
- Mask Repaint Module: This module refines coarse, user-defined masks into spatially accurate ones by exploiting geometric correspondence between the current image latents and reference features.
- Latent Mixture: Instead of simple averaging, this module fuses expert outputs using a learned, timestep-dependent gate to ensure a coherent final image.
- Progressive Curriculum Training: The model is trained first on simple, single-task data before graduating to complex multi-condition samples, preventing "mode collapse" and encouraging expert specialization.
Datasets
The authors utilized several high-quality sources to train and evaluate CARE-Edit:
- Training: MagicBrush, OmniEdit, UNO (subset), and AnyEdit.
- Curated Data: They created a 20K multi-paired corpus from Subjects200K, featuring diverse objects and humans with precise masks and reference images.
- Evaluation Benchmarks: EMU-Edit (instruction-based), MagicBrush (region-based), and DreamBench++ (subject-driven contextual editing).
Key Insights and Ideas
- Timestep Matters: One of the most interesting findings is that the "Base" expert remains active throughout the process to maintain structure, while other experts (like Mask or Reference) spike in activity at specific stages of the diffusion trajectory.
- Disentanglement via Routing: By allowing tokens to bypass irrelevant experts, the model effectively "decouples" style from structure. For instance, the Reference expert can inject artistic textures without distorting the animal's original silhouette, which is protected by the Base and Mask experts.
- The "Base" Anchor: The Base expert acts as a "global anchor," preventing the specialized experts from making changes that would look out of place in the overall context of the image.
Key Results
CARE-Edit consistently outperformed state-of-the-art unified editors:
- Quantitative: On the DreamBench++ benchmark, CARE-Edit achieved the highest scores across all metrics (DINO-I, CLIP-I, and CLIP-T) for both single-object and multiple-object settings, outperforming models like OmniGen2.
- Qualitative: Visual results show significantly cleaner boundaries and better identity preservation. In "subject-driven" tasks (e.g., putting a specific toy in a new scene), CARE-Edit avoided the "hallucinated" artifacts and warped boundaries seen in competing models like ACE++.
- Efficiency: Despite its superior performance, CARE-Edit was trained on significantly less data (~120K triplets) compared to some competitors that use millions of samples.
Limitations and Future Work
The authors acknowledge a few areas for improvement:
- Hyperparameter Sensitivity: The model introduces new hyperparameters, such as the
top-Kvalue for routing, which require careful tuning. - Expert Scalability: While the current four experts cover the most common editing tasks, the authors suggest that future work could involve "dynamic expert loading," allowing the model to plug in new specialized experts for even broader edit types (e.g., depth or pose control) without retraining the entire system.
Summary updated