TL;DR
VitronEdit is a universal framework that unifies image, video, and 3D scene editing within a single model. By using a shared sequence representation and geometry-aware post-training, it successfully transfers semantic editing knowledge from data-rich 2D modalities to data-scarce 3D tasks, achieving state-of-the-art multi-view consistency.
What Problem Does This Paper Address?
Instruction-driven visual editing has seen massive leaps in images and videos, thanks to the abundance of paired training data. However, 3D scene editing remains a "data-poor" relative. High-quality, multi-view 3D editing datasets are scarce, making it difficult to train models that can modify a scene while keeping it geometrically consistent across different viewpoints.
Current approaches often treat these modalities in silos. The authors of VitronEdit ask a fundamental question: Can we learn image, video, and 3D editing jointly? The goal is to allow the model to learn what to edit (semantics) from images and videos, and how to keep those edits consistent (geometry) from limited 3D data.
Methods
VitronEdit treats all three modalities as conditional frame sequences.
- Images are static sequences.
- Videos are temporal sequences.
- 3D scenes are multi-view sequences following a camera trajectory.
Architecture
The system (visualized in Figure 3) uses a Multimodal Large Language Model (MLLM), specifically Qwen2.5-VL, to process text instructions and visual references. This is paired with a Multi-Modal Diffusion Transformer (MMDiT) backbone (HunyuanVideo-1.5) and a video VAE. To distinguish between tasks, the authors introduce learnable modality tokens ([IMG], [VID], [3D]).
Training Strategy
- Progressive Multimodal Training: The model is trained in stages, starting with images, then adding videos, and finally incorporating 3D data.
- Geometry-Aware Post-Training: To enforce structural consistency, the authors use a frozen 3D decoder as a "teacher." This teacher extracts depth maps and camera poses from the model's outputs, providing a supervisory signal that ensures the edited frames actually make sense in a 3D space.
Datasets
The authors curated a massive training set of approximately 2.53 million samples:
- Image: AnyEdit, NHR-Edit, and AnyInsertion (~2M samples).
- Video: OpenVE and Kiwi-Edit (~500K samples).
- 3D: Nano3D and DL3DV.
- New Benchmark: The authors introduced a self-built multiview 3D scene editing dataset (8K samples) to address the lack of high-quality 3D data with significant viewpoint variation.
Key Insights and Ideas
- The Division of Labor: The authors found a clear split in roles. Image and video training primarily transfer editing semantics (e.g., knowing what a "jade carving" looks like), while 3D supervision primarily improves multi-view consistency.
- Cross-Modal Regularization: Surprisingly, applying geometric supervision only to 3D samples yielded the best overall results. The geometric knowledge learned from 3D data acted as a regularizer that improved image and video editing performance through the shared backbone, even though those modalities received no direct geometric supervision.
- Sequence Replication: Standard video VAEs often discard high-frequency details due to temporal compression. To fix this for 3D views (where adjacent frames can look very different), the authors replicated each view four times in the sequence, significantly boosting reconstruction quality (PSNR) and detail preservation.
Key Results
VitronEdit demonstrates impressive versatility across all three domains, as shown in the qualitative results in Figure 1.
- 3D Editing: VitronEdit outperformed dedicated 3D baselines like EditSplat and GaussianEditor, particularly in 3D Consistency (74.34 vs. 66.30 for EditSplat) and Photometric Loss.
- Image/Video Editing: While specialized models sometimes preserve fine-grained source details better, VitronEdit remains highly competitive in semantic alignment and instruction following, proving that a unified model doesn't have to sacrifice 2D quality for 3D capability.
Limitations and Future Work
- Modality Imbalance: The current geometric objectives don't benefit all modalities equally; for example, object motion in videos can sometimes conflict with static geometric priors.
- Static vs. Dynamic: The 3D component currently focuses on static scenes. The authors suggest that extending this to dynamic 3D or 4D scene editing is a major next step.
- Editing Locality: In some cases, such as the "snowy bonsai" example in Figure 6, the model modifies surrounding regions more broadly than requested, indicating a need for better control over edit locality.
Summary updated