TL;DR
TRACE is a novel framework for sequential image editing that prevents "instruction drift" by separating desired attributes (like color) from their visual carriers (the object's surface). By tracking active goals and executing local appearance operations, it ensures that new edits—such as changing a cup's material to metal—don't accidentally undo previous successes, like a prior request to make that cup orange.
What Problem Does This Paper Address?
Sequential image editing allows users to build a result through a conversation (e.g., "Make the cup orange," then "Change its material to metal"). However, current AI editors often suffer from a "forgetting" problem. Because an image is a flat collection of pixels, a model might successfully change a cup's material to metal but, in doing so, revert its color back to the original white, effectively "undoing" the first turn.
The core challenge is that attributes like color and material occupy the same pixels. Standard methods that try to "remember" by retrieving previous images often fail because copying pixels from an old image restores the old color but also restores the old, obsolete material. TRACE addresses this by treating attributes as independent operations that can be reapplied to new surfaces as the edit progresses.
Methods
TRACE functions as an "attribute executor" wrapped around a frozen image editor (like FLUX.2). It consists of three main stages:
- Active-Goal Binding: A text-only interpreter tracks the "state" of the conversation. It identifies which requirements are still "live" (e.g., the cup should be orange) and which have been superseded by new instructions.
- Local Attribute Execution: This is the heart of the system. Instead of letting the diffusion model regenerate everything, TRACE uses a parametric color operator in the Oklab color space. It calculates the color "displacement" needed to reach the target color from a historical reference and applies that displacement to the current object's pixels.
- If the current object lacks the requested material (e.g., it's still plastic when it should be metal), TRACE generates a corrective image to provide a new "carrier" surface while preserving the rest of the scene.
- Complete-Request Selection: A Vision-Language Model (VLM) acts as a "checker." It evaluates candidate images to ensure they satisfy the current request and all previous active goals. It also performs tonal calibration to ensure color changes don't "wash out" natural highlights and shadows.
Datasets
The authors evaluated TRACE using several benchmarks:
- EdiVal: A benchmark for multi-turn editing consisting of 572 sequences.
- MagicBrush: A large-scale dataset of real-world image editing instructions (1,053 independent and 535 multi-turn sessions).
- COCO120: A newly curated set of 120 photographs used to test how well the system transfers to diverse, real-world objects and scenes.
Key Insights and Ideas
- Attribute vs. Carrier: The most important "aha" moment is that an attribute (the "what") and its carrier (the "where") must be bound separately. A mask tells you where to edit, but it doesn't tell you how to reconcile a new material with an old color.
- The Power of Local Execution: By applying edits as local operations rather than global regenerations, TRACE significantly improves scene preservation. It avoids the "background drift" common in sequential editing where the environment subtly changes with every turn.
- Target-Blind Reading: To verify if a material change was successful without being biased by the object's color, the system uses a "target-blind" observer—a VLM that describes the object's material while the specific color goal is withheld from the prompt.
Key Results
- Joint Recovery: In a controlled experiment on color-material conversations, TRACE achieved a joint success rate (keeping both attributes correct) of 96/108, vastly outperforming the raw editor's 62/108.
- Benchmark Performance: On the EdiVal benchmark, TRACE achieved the highest final instruction-following score (31.12%) and the best overall consistency score among compared controllers.
- Qualitative Superiority: As shown in Figure 3, TRACE's tonal calibration preserves fine details (like the dark markings on a bird) that are often lost or "tinted" by standard color-transfer methods.
Limitations and Future Work
- Scope of Attributes: The current implementation focuses heavily on color and material. The authors suggest that extending this "target-carrier separation" to more complex attributes like shape, pose, and lighting is a promising future direction.
- VLM Dependency: The system's success is partially tied to the accuracy of the VLM "checker." If the VLM fails to perceive a subtle attribute, the system may reject a valid edit.
- Computational Cost: Running multiple VLM checks and potentially multiple diffusion draws per turn increases the inference time compared to a simple single-pass editor.
Summary updated