CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing

Yucheng Wang, Zedong Wang, Yuetong Wu, Yue Ma, Dan Xu

TL;DR

CARE-Edit introduces a "Condition-Aware Routing of Experts" framework that solves task interference in unified image editors. By dynamically dispatching image tokens to specialized experts (Text, Mask, Reference, and Base) based on the current diffusion timestep and input signals, it achieves high-fidelity results in complex tasks like subject replacement and style transfer without the artifacts common in static models.

What Problem Does This Paper Address?

Current unified diffusion-based image editors (like ControlNet or OmniControl) often struggle when multiple conditioning signals—such as a text prompt, a spatial mask, and a reference image—are provided simultaneously.

The authors identify a core issue: static fusion. Most models simply concatenate these signals or use additive adapters, forcing a single shared backbone to process everything at once. This leads to "task interference," where signals conflict. For example, a text prompt might override a mask boundary (causing color bleeding), or a reference image might cause "identity drift" where the subject doesn't quite match the original. Furthermore, the importance of these signals changes during the generation process (e.g., layout matters early on, while fine details matter later), but static models can't adapt to this temporal shift.

Methods

The researchers propose CARE-Edit, which moves away from a "one-size-fits-all" backbone toward a Mixture-of-Experts (MoE) architecture. Key components include:

  • Heterogeneous Experts: The model employs four specialized, lightweight adapters (experts):
  • Text Expert: Handles semantic reasoning and object synthesis.
  • Mask Expert: Focuses on spatial precision and boundary refinement.
  • Reference Expert: Manages identity and style-consistent transformations.
  • Base Expert: Ensures global coherence and background preservation.
  • Latent-Attention Router: A lightweight module that uses sparse top-K selection to dynamically assign each diffusion token to the most relevant experts. This routing is "condition-aware," meaning it looks at the prompt, mask, and reference features to decide which expert should take the lead for a specific part of the image.
  • Mask Repaint Module: This module refines coarse, user-defined masks into spatially accurate ones by exploiting geometric correspondence between the current image latents and reference features.
  • Latent Mixture: Instead of simple averaging, this module fuses expert outputs using a learned, timestep-dependent gate to ensure a coherent final image.
  • Progressive Curriculum Training: The model is trained first on simple, single-task data before graduating to complex multi-condition samples, preventing "mode collapse" and encouraging expert specialization.

Datasets

The authors utilized several high-quality sources to train and evaluate CARE-Edit:

  • Training: MagicBrush, OmniEdit, UNO (subset), and AnyEdit.
  • Curated Data: They created a 20K multi-paired corpus from Subjects200K, featuring diverse objects and humans with precise masks and reference images.
  • Evaluation Benchmarks: EMU-Edit (instruction-based), MagicBrush (region-based), and DreamBench++ (subject-driven contextual editing).

Key Insights and Ideas

  • Timestep Matters: One of the most interesting findings is that the "Base" expert remains active throughout the process to maintain structure, while other experts (like Mask or Reference) spike in activity at specific stages of the diffusion trajectory.
  • Disentanglement via Routing: By allowing tokens to bypass irrelevant experts, the model effectively "decouples" style from structure. For instance, the Reference expert can inject artistic textures without distorting the animal's original silhouette, which is protected by the Base and Mask experts.
  • The "Base" Anchor: The Base expert acts as a "global anchor," preventing the specialized experts from making changes that would look out of place in the overall context of the image.

Key Results

CARE-Edit consistently outperformed state-of-the-art unified editors:

  • Quantitative: On the DreamBench++ benchmark, CARE-Edit achieved the highest scores across all metrics (DINO-I, CLIP-I, and CLIP-T) for both single-object and multiple-object settings, outperforming models like OmniGen2.
  • Qualitative: Visual results show significantly cleaner boundaries and better identity preservation. In "subject-driven" tasks (e.g., putting a specific toy in a new scene), CARE-Edit avoided the "hallucinated" artifacts and warped boundaries seen in competing models like ACE++.
  • Efficiency: Despite its superior performance, CARE-Edit was trained on significantly less data (~120K triplets) compared to some competitors that use millions of samples.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Hyperparameter Sensitivity: The model introduces new hyperparameters, such as the top-K value for routing, which require careful tuning.
  • Expert Scalability: While the current four experts cover the most common editing tasks, the authors suggest that future work could involve "dynamic expert loading," allowing the model to plug in new specialized experts for even broader edit types (e.g., depth or pose control) without retraining the entire system.

Summary updated

Abstract

Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs photometric). In particular, prevalent ControlNet and OmniControl variants combine multiple conditioning signals (e.g., text, mask, reference) via static concatenation or additive adapters which cannot dynamically prioritize or suppress conflicting modalities, thus resulting in artifacts like color bleeding across mask boundaries, identity or style drift, and unpredictable behavior under multi-condition inputs. To address this, we propose Condition-Aware Routing of Experts (CARE-Edit) that aligns model computation with specific editing competencies. At its core, a lightweight latent-attention router assigns encoded diffusion tokens to four specialized experts--Text, Mask, Reference, and Base--based on multi-modal conditions and diffusion timesteps: (i) a Mask Repaint module first refines coarse user-defined masks for precise spatial guidance; (ii) the router applies sparse top-K selection to dynamically allocate computation to the most relevant experts; (iii) a Latent Mixture module subsequently fuses expert outputs, coherently integrating semantic, spatial, and stylistic information to the base images. Experiments validate CARE-Edit's strong performance on contextual editing tasks, including erasure, replacement, text-driven edits, and style transfer. Empirical analysis further reveals task-specific behavior of specialized experts, showcasing the importance of dynamic, condition-aware processing to mitigate multi-condition conflicts.
BibTeX
@inproceedings{wang2026care,
  author = {Yucheng Wang and Zedong Wang and Yuetong Wu and Yue Ma and Dan Xu},
  title = {{CARE-Edit: Condition-Aware Routing of Experts for Contextual Image Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=wgw9WrgENv},
}
Figure