VideoEditor-MoE: Scaling Complex Video Editing with Mixture of Experts

Xiao Cao, Fuchen Long, Yansong Qu, Wen Xiao, Xin Zhang, Xinlong Sun, Xi Chen, Zhiyong HuangarXiv 2026 · arXiv

TL;DR

VideoEditor-MoE is the first open-source Mixture-of-Experts (MoE) framework designed specifically for complex video editing. By introducing a novel MoE-Attention mechanism alongside sparse MLP layers, the researchers scaled a 5B base model up to 56B parameters while maintaining the computational efficiency of an 8B active parameter model, significantly improving the fulfillment of multi-part editing instructions.

What Problem Does This Paper Address?

Real-world video editing requests are rarely simple. A user might ask to "turn the car blue while replacing the sky with a sunset." This requires the model to track multiple objects, maintain temporal consistency, and coordinate several distinct modifications simultaneously.

While scaling "dense" models (where every parameter is used for every calculation) improves performance, it leads to a massive spike in computational costs. Traditional Mixture-of-Experts (MoE) approaches solve this by only activating a subset of "experts" per token, but they typically only scale the Multi-Layer Perceptron (MLP) blocks. In video, however, the attention mechanism is the engine of global coordination across space and time. The authors argue that scaling only the MLPs leaves a bottleneck in semantic understanding, making it difficult to handle complex, compositional editing tasks.

Methods

The authors introduce VideoEditor-MoE, built upon a 5B-parameter Diffusion Transformer (DiT) base. The architecture features two primary innovations:

  1. MoE-Attention: Unlike standard MoE which targets MLPs, MoE-Attention expands the capacity of the self-attention mechanism. It uses shared queries but routes tokens to specific key-value (KV) experts. Every query attends to a global pool of these routed KV pairs, preserving the model's ability to relate distant parts of the video while allowing for much richer semantic modeling.
  2. Scaling Strategy: They developed two variants: 14B-A8B and 56B-A8B (where "A8B" denotes ~8B activated parameters per token). To stabilize training at these scales, they identified that standard Adam optimization with coupled $L_2$ regularization caused "expert-weight shrinkage." They switched to AdamW (decoupled weight decay), which prevents the optimizer from suppressing expert outputs.
The VideoEditor-MoE framework alternates between dense and MoE DiT blocks, utilizing a specialized MoE-Attention mechanism.

The team also compared their approach against two internal baselines: Explicitly Guided MoE (which uses human-defined categories like "object removal" to route experts) and a standard MLP-MoE (which keeps attention dense).

Datasets

The model was initialized from a pretrained 5B video editing DiT. Training involved clips of 49 frames at $480 \times 832$ resolution. For evaluation, the authors compared VideoEditor-MoE against several state-of-the-art methods, including Kiwi-Edit, MiniMax-H3, and CoinVE-Edit. Quantitative evaluation was performed using a protocol involving Qwen2.5-27B as an impartial judge to score edit execution and fulfillment across video frames.

Key Insights and Ideas

  • Routing is Content-Driven, Not Category-Driven: One of the most surprising findings was that the learned routers did not align with human-defined task categories (like "style transfer"). Instead, routing decisions were much more sensitive to the actual visual content of the video and the diffusion noise level.
  • Spatial Semantic Sensitivity: The experts aren't just random math blocks; they specialize spatially. Analysis showed that routing weights for specific experts followed moving objects (like a person walking) across frames, suggesting the experts learn to handle specific semantic features.
Visualizing how routing weights follow a moving object, demonstrating sensitivity to semantic content.
  • Attention Scaling is Vital: The gap between the MLP-MoE baseline and VideoEditor-MoE highlights that for video, expanding the "memory" of the attention mechanism (via KV experts) is just as important as expanding the transformation capacity of the MLPs.

Key Results

VideoEditor-MoE-56B-A8B outperformed all baselines across three primary metrics:

  • Edit Execution Rate: Achieved 94.70%, significantly higher than the best existing method, CoinVE-Edit (87.99%).
  • Edit Fulfillment: Scored 94.70%, proving it can realize complex instructions more completely than dense or MLP-only MoE models.
  • Video Quality: Reached a score of 77.27, indicating better temporal consistency and preservation of unedited content.

Qualitatively, the model excels at "compositional" edits where other models fail. For instance, in a request to add bananas to a kitchen while applying a watercolor style, VideoEditor-MoE successfully integrated both, whereas baselines often omitted one of the requests.

Qualitative comparison showing VideoEditor-MoE successfully handling multiple simultaneous editing requests.

Limitations and Future Work

While VideoEditor-MoE represents a leap forward, the authors note a few areas for growth:

  • Evaluation Bias: The evaluation protocol used a fixed inference configuration for all models, which might favor the specific strengths of the VideoEditor-MoE architecture.
  • Computational Overhead: Although active parameters are kept low, the total memory footprint of a 56B model remains a challenge for consumer-grade hardware.
  • Future Directions: The authors suggest that the success of MoE-Attention in editing could be extended to base video generation models to improve their adherence to complex prompts from the start.

Summary updated

Abstract

Complex video editing requires understanding complex instructions while jointly fulfilling multiple editing requests. Meeting these demands through dense model scaling substantially increases computational cost. We introduce VideoEditor-MoE, the first open-source mixture-of-experts framework for video editing. Beyond sparse MLP-MoE layers, MoE-Attention expands attention capacity through routed key--value experts while preserving global interactions among video tokens. Initialized from a 5B video editing model, VideoEditor-MoE scales to 14B and 56B parameters, with approximately 8B parameters activated per token in both variants. Comparisons with existing methods and two complementary baselines, Explicit-MoE and MLP-MoE, assess its benefits over explicit expert guidance and MLP-only expert architectures. Further analyses demonstrate that all experts remain active and reveal that learned routing responds more strongly to sample content and diffusion noise levels than to human-defined editing categories. Spatial analysis further indicates sensitivity to object-level semantic features. We also identify decoupled weight decay as a strategy for stabilizing expert scaling.
BibTeX
@misc{cao2026videoeditor,
  author = {Xiao Cao and Fuchen Long and Yansong Qu and Wen Xiao and Xin Zhang and Xinlong Sun and Xi Chen and Zhiyong Huang},
  title = {{VideoEditor-MoE: Scaling Complex Video Editing with Mixture of Experts}},
  year = {2026},
  eprint = {2611.03043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03043},
}
Figure