TL;DR
VideoEditor-MoE is the first open-source Mixture-of-Experts (MoE) framework designed specifically for complex video editing. By introducing a novel MoE-Attention mechanism alongside sparse MLP layers, the researchers scaled a 5B base model up to 56B parameters while maintaining the computational efficiency of an 8B active parameter model, significantly improving the fulfillment of multi-part editing instructions.
What Problem Does This Paper Address?
Real-world video editing requests are rarely simple. A user might ask to "turn the car blue while replacing the sky with a sunset." This requires the model to track multiple objects, maintain temporal consistency, and coordinate several distinct modifications simultaneously.
While scaling "dense" models (where every parameter is used for every calculation) improves performance, it leads to a massive spike in computational costs. Traditional Mixture-of-Experts (MoE) approaches solve this by only activating a subset of "experts" per token, but they typically only scale the Multi-Layer Perceptron (MLP) blocks. In video, however, the attention mechanism is the engine of global coordination across space and time. The authors argue that scaling only the MLPs leaves a bottleneck in semantic understanding, making it difficult to handle complex, compositional editing tasks.
Methods
The authors introduce VideoEditor-MoE, built upon a 5B-parameter Diffusion Transformer (DiT) base. The architecture features two primary innovations:
- MoE-Attention: Unlike standard MoE which targets MLPs, MoE-Attention expands the capacity of the self-attention mechanism. It uses shared queries but routes tokens to specific key-value (KV) experts. Every query attends to a global pool of these routed KV pairs, preserving the model's ability to relate distant parts of the video while allowing for much richer semantic modeling.
- Scaling Strategy: They developed two variants: 14B-A8B and 56B-A8B (where "A8B" denotes ~8B activated parameters per token). To stabilize training at these scales, they identified that standard Adam optimization with coupled $L_2$ regularization caused "expert-weight shrinkage." They switched to AdamW (decoupled weight decay), which prevents the optimizer from suppressing expert outputs.
The team also compared their approach against two internal baselines: Explicitly Guided MoE (which uses human-defined categories like "object removal" to route experts) and a standard MLP-MoE (which keeps attention dense).
Datasets
The model was initialized from a pretrained 5B video editing DiT. Training involved clips of 49 frames at $480 \times 832$ resolution. For evaluation, the authors compared VideoEditor-MoE against several state-of-the-art methods, including Kiwi-Edit, MiniMax-H3, and CoinVE-Edit. Quantitative evaluation was performed using a protocol involving Qwen2.5-27B as an impartial judge to score edit execution and fulfillment across video frames.
Key Insights and Ideas
- Routing is Content-Driven, Not Category-Driven: One of the most surprising findings was that the learned routers did not align with human-defined task categories (like "style transfer"). Instead, routing decisions were much more sensitive to the actual visual content of the video and the diffusion noise level.
- Spatial Semantic Sensitivity: The experts aren't just random math blocks; they specialize spatially. Analysis showed that routing weights for specific experts followed moving objects (like a person walking) across frames, suggesting the experts learn to handle specific semantic features.
- Attention Scaling is Vital: The gap between the MLP-MoE baseline and VideoEditor-MoE highlights that for video, expanding the "memory" of the attention mechanism (via KV experts) is just as important as expanding the transformation capacity of the MLPs.
Key Results
VideoEditor-MoE-56B-A8B outperformed all baselines across three primary metrics:
- Edit Execution Rate: Achieved 94.70%, significantly higher than the best existing method, CoinVE-Edit (87.99%).
- Edit Fulfillment: Scored 94.70%, proving it can realize complex instructions more completely than dense or MLP-only MoE models.
- Video Quality: Reached a score of 77.27, indicating better temporal consistency and preservation of unedited content.
Qualitatively, the model excels at "compositional" edits where other models fail. For instance, in a request to add bananas to a kitchen while applying a watercolor style, VideoEditor-MoE successfully integrated both, whereas baselines often omitted one of the requests.
Limitations and Future Work
While VideoEditor-MoE represents a leap forward, the authors note a few areas for growth:
- Evaluation Bias: The evaluation protocol used a fixed inference configuration for all models, which might favor the specific strengths of the VideoEditor-MoE architecture.
- Computational Overhead: Although active parameters are kept low, the total memory footprint of a 56B model remains a challenge for consumer-grade hardware.
- Future Directions: The authors suggest that the success of MoE-Attention in editing could be extended to base video generation models to improve their adherence to complex prompts from the start.
Summary updated