TrAME: Trajectory-Anchored Multi-View Editing for Text-Guided 3D Gaussian Manipulation

Chaofan Luo, Donglin Di, Xun Yang, Yongjia Ma, Zhou Xue, Wei Chen, Xiaofei Gou, Yebin LiuIEEE Trans. Multimedia 2025 · DOI · OpenReview

TL;DR

This paper introduces TrAME, a novel framework for text-guided 3D scene editing that significantly improves multi-view consistency. By tightly coupling 2D view editing with 3D updates through a Trajectory-Anchored Scheme (TAS) and a View-Consistent Attention Control (VCAC) module, TrAME prevents error accumulation and produces more coherent and realistic edits.

What Problem Does This Paper Address?

Editing 3D scenes using text prompts has become increasingly powerful, but a major hurdle remains: maintaining consistency across multiple views of the edited scene. When you edit a 3D object or scene, you're essentially changing its underlying representation. If this change isn't applied consistently from every possible viewing angle, the result can be jarring visual artifacts, like flickering or distorted geometry, making the edited scene look unnatural and broken. Current methods fall into two camps: optimization-based (which can suffer from sub-optimal quality and misalignment) and reconstruction-based (which struggle with inter-view consistency due to their independent processing of each view). This paper aims to bridge this gap by developing a method that ensures robust multi-view consistency throughout the editing process.

Methods

TrAME introduces two core innovations:

  • Trajectory-Anchored Scheme (TAS): This is the heart of the progressive editing strategy. TAS tightly couples the 2D image editing process with the 3D scene update process. It works iteratively: first, 2D views are edited based on the text prompt. Then, these edited views are used to update the 3D representation (specifically, 3D Gaussian Splatting or 3DGS). Crucially, the rendered views from the updated 3DGS are then fed back to correct any inconsistencies that arose during the 2D editing. This "rendering-in-the-loop" approach prevents errors from accumulating over time. The paper also theoretically connects optimization-based and reconstruction-based methods, showing that optimization methods can be viewed as special cases of reconstruction, providing a unified perspective for designing editing strategies.
  • View-Consistent Attention Control (VCAC): This module is designed to enhance 3D consistency during the 2D view editing phase. It employs a dual-branch editing strategy:
  • Source Branch: Uses the original, unedited views as a reference.
  • Target Branch: Performs the actual editing. VCAC injects self-attention queries from the source branch into the target branch during early diffusion stages to maintain structural consistency. For semantic consistency, it uses "KV propagation" and "KV reference" mechanisms. KV propagation shares self-attention keys and values within a local context (e.g., consecutive frames), while KV reference allows keyframes from different contexts to mutually inform each other, preventing semantic drift.

Datasets

The paper evaluates its method on the Instruct-NeRF2NeRF (IN2N) and Mip-NeRF-360 datasets. These are standard benchmarks for 3D scene editing and representation tasks. No new datasets were introduced.

Key Insights and Ideas

  • Bridging Optimization and Reconstruction: The theoretical analysis showing optimization-based methods as a subset of reconstruction-based methods is a significant conceptual contribution. It provides a unified framework for understanding and designing 3D editing techniques.
  • The Power of "Rendering-in-the-Loop": The core idea of using the updated 3D representation to feedback and correct 2D edits is a clever way to enforce consistency. It directly addresses the problem of error propagation in iterative editing.
  • Attention Mechanisms for Cross-View Consistency: Leveraging self-attention not just within a single view but across multiple views (via query injection and KV propagation/reference) is a powerful way to enforce both structural and semantic alignment.
  • Annealing Timestep Schedule for SDS: The paper highlights the importance of an annealing timestep schedule for Score Distillation Sampling (SDS) to avoid issues like over-saturation and over-smoothing, a known challenge in SDS-based methods.

Key Results

  • Quantitative: TrAME achieves superior performance compared to state-of-the-art methods on various metrics. For reconstruction-based methods, it shows significantly higher consistency between 2D edits and 3D scene modifications, as evidenced by improved PSNR, SRE, RMSE, and CLIP Image-to-Image Similarity scores (e.g., a 102.449 PSNR with TAS and VCAC vs. 87.860 for IN2N). In terms of text-to-3D alignment, TrAME also leads in CLIP Directional Score (e.g., 0.199 ±0.054 with ViT-B/16).
  • Qualitative: Visual results demonstrate TrAME's ability to produce naturalistic and consistent edits across different views. Ablation studies show that both TAS and VCAC contribute significantly to this improved consistency, with the ablated versions exhibiting more artifacts and inconsistencies. The method excels in preserving intricate details and textures, even in complex scenes.

Limitations and Future Work

  • Complex Deformations: The paper acknowledges that TrAME, like existing baselines, struggles with edits that require significant object deformation or displacement. This is attributed to the limitations of 2D image editing models and insufficient integration of explicit 3D geometry.
  • Future Directions: The authors plan to leverage explicit 3D Gaussian Splatting geometry more deeply to incorporate multi-level hierarchical editing guidance, aiming to facilitate more complex edits. They also note the broader societal impact, highlighting the potential for misuse in creating misleading content.

Summary updated

Abstract

Despite significant strides in the field of 3D scene editing, current methods encounter substantial challenge, particularly in preserving 3D consistency during the multi-view editing process. To tackle this challenge, we propose a progressive 3D editing strategy that ensures multi-view consistency via a Trajectory-Anchored Scheme (TAS) with a dual-branch editing mechanism. Specifically, TAS facilitates a tightly coupled iterative process between 2D view editing and 3D updating, preventing error accumulation yielded from the text-to-image process. Additionally, we explore the connection between optimization-based methods and reconstruction-based methods, offering a unified perspective for selecting superior design choices, supporting the rationale behind the designed TAS. We further present a tuning-free View-Consistent Attention Control (VCAC) module that leverages cross-view semantic and geometric reference from the source branch to yield aligned views from the target branch during the editing of 2D views. To validate the effectiveness of our method, we analyze 2D examples to demonstrate the improved consistency with the VCAC module. Extensive quantitative and qualitative results in text-guided 3D scene editing clearly indicate that our method can achieve superior editing quality compared with state-of-the-art 3D scene editing methods.
BibTeX
@article{luo2025trame,
  author = {Chaofan Luo and Donglin Di and Xun Yang and Yongjia Ma and Zhou Xue and Wei Chen and Xiaofei Gou and Yebin Liu},
  title = {{TrAME: Trajectory-Anchored Multi-View Editing for Text-Guided 3D Gaussian Manipulation}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3557618},
}
Figure