DecoEdit: Beyond Flat Control for 3D-Aware Unified Image Editing

Xiaotang Gai, Jinbo Xing, Zijie Meng, Jiayi Yi, Jimin Xiao, Yixuan Qian, Yu Liu, Chaojie Mao, Zuozhu LiuarXiv 2026 · arXiv

TL;DR

DecoEdit is a unified image editing framework that moves beyond "flat" 2D controls (like masks and text) to enable precise 3D manipulations, such as rotating or scaling objects while perfectly preserving their identity. By decoupling 3D geometric guidance from high-fidelity texture preservation and training on a massive new 1.6-million-sample dataset, it outperforms leading models like Seedream 5.0 in geometric accuracy and visual consistency.

What Problem Does This Paper Address?

Most modern image editors are "flat." If you want to change an image, you usually provide a text prompt, a 2D bounding box, or a binary mask. While these work for restyling or simple insertions, they struggle with 3D-aware requests like, "Relocate this chair to the left and rotate it 45 degrees toward the camera."

The core challenge is appearance-geometry entanglement. When a model tries to rotate an object in 2D pixel space, it often "smears" the texture, losing fine details like logos, wood grains, or intricate patterns. Existing solutions either require slow, expensive per-instance fine-tuning or produce blurry, distorted results that don't respect the 3D structure of the scene. Furthermore, there has been a lack of large-scale datasets that jointly annotate 1D (text), 2D (spatial), and 3D (volumetric) instructions.

Methods

The authors introduce DecoEdit, an all-in-one framework built on a Mixture-of-Transformers (MoT) architecture. It processes four distinct types of input tokens:

  1. Semantic Tokens: Text instructions and image references.
  2. 2D Spatial Tokens: Target coordinates (bounding boxes).
  3. Source Appearance Tokens: High-frequency textures from the original object.
  4. 3D Target Pose Tokens: Structural guidance derived from SAM-3D.

The "secret sauce" of DecoEdit is its Appearance-Geometry Decoupled Strategy. Instead of forcing the model to guess a new 3D pose from 2D pixels, the system uses a 3D proxy (SAM-3D) to render a "pose scaffold." This scaffold provides the geometric "bones" of the edit. Simultaneously, the original source image is routed through a separate path to "lock in" the identity and textures. The model then learns to "paint" the high-fidelity textures from the source onto the new 3D scaffold.

Datasets

The paper introduces two major resources:

  • DecoDataset: A massive training set of 1.63 million samples. It was built using an automated pipeline that takes 2D image pairs and "lifts" them into 3D using reconstruction tools (SAM-3D and MoGe). It covers 42 different control subtasks, ranging from simple color changes to complex 3D rotations.
  • DecoBench: A comprehensive evaluation suite of 261 testing cases. Unlike previous benchmarks, it specifically targets multi-dimensional editing, including "physically violating" edge cases (e.g., objects defying gravity) to test the limits of the model’s controllability.

Key Insights and Ideas

  • Decoupling is Essential: The authors found that trying to handle geometry and texture in a single stream leads to catastrophic detail degradation. By treating the 3D pose as a "scaffold" and the source image as a "texture map," they solve the entanglement problem.
  • The Power of Unified Training: Ablation studies showed that removing any one dimension (1D, 2D, or 3D) from the training mixture hurt the others. For example, dropping 1D text supervision actually degraded the model's ability to follow 2D spatial constraints, suggesting that the model learns a more robust internal representation when forced to handle all dimensions simultaneously.
  • Automated 3D Curation: The ability to turn "flat" 2D image datasets into 3D-aware training data via automated reconstruction (the "reconstruct-render-align loop") is a scalable way to overcome the data scarcity that has plagued 3D vision-language research.

Key Results

DecoEdit achieves state-of-the-art (SOTA) performance across the board:

  • Quantitative: On the DecoBench macro score, DecoEdit scored 8.97, significantly higher than the strongest open-source baseline, SenseNova (8.01), and leading closed-source models like Seedream 5.0 (8.58) and GPT-Image-1.5 (8.71).
  • Qualitative: In 3D orientation tasks, DecoEdit successfully preserves intricate details (like text on a motorcycle or the specific pattern of a mannequin) that other models either blur out or replace with generic textures.
  • Human Preference: In a blind study, human raters ranked DecoEdit first (70.25) over Seedream 5.0 (66.31) and Gemini 2.5 Flash (42.31).

Limitations and Future Work

  • Dependency on 3D Proxies: The quality of the 3D edit is currently bounded by the accuracy of the initial 3D reconstruction (SAM-3D). If the proxy geometry is wrong, the final edit will be geometrically flawed.
  • Computational Intensity: The model uses a dual-stream 14B parameter architecture. While powerful, this requires significant VRAM, making it less accessible for consumer-grade hardware without optimization.
  • Future Directions: The authors suggest extending this framework to video editing, where maintaining 3D consistency across frames is an even greater challenge.

For more details, see the full paper: "DecoEdit: Beyond Flat Control for 3D-Aware Unified Image Editing" (arXiv:2611.19121v1).

Summary updated

Abstract

While recent advances in multimodal image editing have achieved impressive results, current frameworks remain largely confined to ``flat" 2D controls (e.g., text prompts, bounding boxes, or masks). Precise 3D-aware geometric manipulation -- such as coordinating an object's location, physical scale, and 3D orientation while strictly preserving its fine-grained identity -- remains an elusive challenge, typically suffering from severe detail degradation or requiring expensive per-instance fine-tuning. In this paper, we present DecoEdit, an all-in-one image editing framework that goes beyond flat control by seamlessly integrating 1D textual, 2D spatial, and 3D volumetric inputs under a unified architecture. First, to power this unified model, we establish a data curation pipeline and curate a large-scale dataset, namely DecoDataset, containing 1.6 million samples, encompassing diverse multi-dimensional (1D/2D/3D) tasks and their complex compositions. Furthermore, we establish a comprehensive image editing benchmark named DecoBench to systematically evaluate multi-task editing performance. Finally, to address the fundamental trade-off between complex spatial transformations and identity preservation, we propose a novel decoupled pipeline: DecoEdit factorizes the editing process by utilizing structural guidance from SAM-3D to anchor the target geometry, scale, and spatial orientation, while concurrently leveraging a detail-preservation path to lock in high-fidelity textures. Extensive experiments demonstrate that DecoEdit achieves state-of-the-art results. Notably, on the challenging 3D spatial pose and orientation control tasks, DecoEdit significantly outperforms leading closed-source models such as Seedream 5.0 in both geometric accuracy and identity consistency.
BibTeX
@misc{gai2026decoedit,
  author = {Xiaotang Gai and Jinbo Xing and Zijie Meng and Jiayi Yi and Jimin Xiao and Yixuan Qian and Yu Liu and Chaojie Mao and Zuozhu Liu},
  title = {{DecoEdit: Beyond Flat Control for 3D-Aware Unified Image Editing}},
  year = {2026},
  eprint = {2611.19121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19121},
}
Figure