Text-to-image diffusion models have become remarkably adept at "rigid" edits—changing a subject's color, swapping a background, or replacing one object with another. However, as noted by Mingdeng Cao and Ligong Han, these models often fail when asked to perform "non-rigid" transformations, such as altering a character's pose, bending an object, or changing a facial expression.
The core challenge is a phenomenon often called "structural lock." To preserve the identity of a real image, most editing frameworks use inversion techniques (like DDIM) to find a noise map that reconstructs the original. This noise map, along with the model's internal attention maps, acts as a rigid skeleton that resists geometric changes. When a user asks to "make the sitting dog run," the model experiences a conflict: the text prompt demands a new layout, but the underlying noise and attention maps demand the old one, leading to distorted artifacts or a failure to move the subject at all.
Decoupling Identity from Layout via Mutual Attention
To break this structural lock, researchers have moved away from strict noise synchronization toward mechanisms that decouple what an object looks like from where it is located.
A landmark approach in this direction is MasaCtrl, introduced by Mingdeng Cao. Instead of standard self-attention, which forces a frame to look only at itself, MasaCtrl uses mutual self-attention. During the denoising process, the model generates two parallel streams: a source stream (the original image) and a target stream (the edited version). The target stream is allowed to "query" the textures and identity features from the source stream's keys and values. This allows the target image to adopt a completely new layout or pose defined by a text prompt while "borrowing" the specific visual details of the source subject.
Similarly, Ligong Han and colleagues proposed SINE, which uses patch-based fine-tuning and model-based guidance to help the diffusion process navigate drastic spatial rearrangements without losing the subject's identity. By focusing on local patches, the model can learn the "essence" of the subject without being tethered to its global coordinates.
Learning Motion Priors from Video
A recurring insight in recent work is that static image datasets do not contain enough information about how objects physically deform. As Yanpeng Sun argues in Video4Edit, a non-rigid edit is essentially a "degenerate temporal process"—a two-frame video showing a start and end state.
By distilling temporal priors from large-scale video models into image editors, researchers can teach models the physics of movement. Mingdeng Cao followed this logic with InstructMove, creating a massive dataset of 6 million triplets by "watching" how things move in videos. This allows the model to understand that "raising an arm" involves specific anatomical articulations rather than just a messy smudge of pixels.
Loosening the Latent Constraints
Another strategy involves surgically relaxing the constraints of the inversion process. Gwanhyeong Koo and the creators of FlexiEdit identified that the "structural lock" is primarily held in the high-frequency components of the latent noise. By selectively refining these high-frequency latents, they allow the model enough flexibility to move limbs or change shapes while keeping the low-frequency global structure (like lighting and background) intact.
In the realm of flow-based models like Stable Diffusion 3 and Flux, methods like FireFlow by Yingying Deng and DirectEdit by Desong Yang attempt to align the denoising path with the inversion trajectory more accurately. These "flow-matching" approaches provide a more linear and predictable path for editing, making it easier to execute significant structural departures from the source image without the "drift" that plagues traditional DDIM inversion.
3D-Aware and Region-Based Guidance
Finally, some researchers argue that 2D pixels alone are insufficient for complex deformations. FlowDrag introduces 3D structural awareness by deforming a 3D mesh to guide 2D pixel movements, ensuring that an object’s geometry remains consistent during a rotation or pose change.
For users who need "surgical" precision, Jun Zhou and the FireEdit team utilize region-aware Vision Language Models (VLMs). By using explicit "region tokens," the model can isolate the specific part of the image that needs to deform—like a single leg or a sleeve—preventing the non-rigid edit from "leaking" and distorting the rest of the scene. This spatial grounding, also explored by Peter Wonka in LASPA, ensures that even when an object undergoes a massive structural overhaul, the global composition remains harmonious.