Pose-Warped Residual Inversion: Editing a Real Person's Pose Without Losing the Photo

Suppose you want to take a real photograph of a person, bend their elbow or turn their torso, and keep everything else as it was: the sweater weave, the face, the background. Generative image models can almost do this. The recipe is inversion: running the image generator backward to recover the random noise that would have produced the photo, then regenerating from that noise with a new instruction. The most faithful current inversion methods reach this fidelity by freezing the image's spatial layout. As a result they cannot move limbs. The opposite family, generators that follow a 3D body model, can move limbs freely but has no way to start from an arbitrary real photo and reproduce it exactly. This note proposes connecting the two. We fit a parametric 3D body model to the photo. That is a low-dimensional mesh whose joint angles and shape coefficients describe a human body; SMPL-X is the standard one. We then use the mesh to carry the inversion's stored correction signals from the source pose to the target pose. Where that transport is physically impossible, because body parts fold over, tear apart, or become newly visible, a local conflict detector switches the stored signal off and lets the generator fill in.

The conflict detector borrows one idea from geometric magnetic frustration, the situation in a triangular lattice where three antiferromagnetic spins cannot all be pairwise opposite. We treat triples of neighbouring image patches the same way. If warping them to the new pose flips their orientation or stretches them far apart, their stored inversion signals cannot all be honoured at once, and that region is labelled "frustrated." This is a borrowed diagnostic, not a physics claim, and the proposal includes the ablation that would show whether it beats a plain visibility mask. The fourth supplied direction, quantum query-complexity lower bounds, has no mechanistic connection here beyond a loose analogy about error accumulation during reconstruction. We set it aside rather than force it in.

Core Claim

Exact-inversion editors fail at pose changes because their stored corrections are tied to the source pixel grid, not because inversion is inherently rigid. DirectEdit records, at every step of a rectified-flow inversion, the difference between the ideal trajectory and the discrete Euler step. Rectified flow is a generator that transports noise to images along nearly straight paths. DirectEdit replays that difference, together with early self-attention values, during editing. This gives background PSNR of 32.63 dB on PIE-Bench with FLUX, against 25.20 dB for the next method, with no extra network evaluations. The same paper states that the recorded noise has a "rigid structural nature" that blocks pose and layout changes.

Our claim is that re-indexing these residuals and attention values through a dense correspondence field derived from the source and target SMPL-X meshes will recover most of DirectEdit's identity and background fidelity under articulated pose edits. Suppressing injection only in geometrically frustrated regions will beat both always-on injection, which causes ghosting, and mesh-visibility masking, which fails on loose clothing. What is new is a pose-aware transport of step-level inversion residuals, plus a local rule for when to drop them. As far as the file shows, no inversion method uses a 3D body prior.

Relation to Prior Work

Two inversion lines supply the machinery. DirectEdit supplies the residual recording we reuse. FreeInv applies the same random 0°/90°/180°/270° latent rotation at each step in both the inversion and reconstruction branches, which averages out drift. It raises FLUX.1 reconstruction PSNR from 14.92 to 29.24 dB. FreeInv is also evidence that geometric transformations applied consistently to both branches do not break inversion; our method relies on the same premise with non-rigid warps. Neither method touches articulated geometry, since FreeInv's transforms are global and rigid.

Symmetric integrators, namely BDIA and O-BELM (Wang et al.), reduce truncation drift and are complementary; we can adopt them without changing the idea. Three results constrain the design. Han Zhang et al. show that small latent perturbations grow exponentially under reverse denoising, so crude warps of the noise alone should be unstable. This is why we transport per-step residuals rather than only the initial noise. Zheng et al. show that local latent edits bleed across seams when the autoencoder decodes them, which motivates a smoothness term in the switching rule. Dong et al. show that high guidance scales destabilise inverted backgrounds, so guidance strength must be held fixed across all compared methods.

On the pose side, GenFusion supplies the correspondence mechanism. It warps features between a canonical T-pose and live poses through SMPL-X barycentric coordinates and renders with a pose-conditioned diffusion model, reaching FVD 176.7 on 4D-Dress against 254.5 for the per-frame Champ baseline. It is feed-forward, however, and never inverts a real image. GauHuman, DreamAvatar, and AvatarFusion similarly anchor 3D representations to body meshes, and share GenFusion's known weaknesses: template bias on loose garments, and sensitivity to errors in the upstream pose fit. Our frustration detector targets exactly the first of these. Sangineto et al. and Guan et al. motivate using a 3D template rather than 2D keypoints. Their evidence is that 2D-only supervision loses geometric stability.

Research Program

The pipeline has four stages. First, fit SMPL-X to the source image and specify the target pose, either as edited joint angles or copied from a reference. Second, invert the source with DirectEdit on a FLUX backbone, conditioned on a rendering of the source mesh (normals and depth) through a pose adapter. We use an off-the-shelf adapter if one fits the backbone; otherwise we train a lightweight adapter on rendered SMPL-X maps from THuman2.1 and MVHumanNet. We store the per-step residuals and early-layer value features.

Third, compute a dense correspondence field at latent resolution. Body pixels map through mesh barycentric coordinates. Background pixels map to themselves. Pixels that become newly visible have no source. We warp every stored residual and value tensor through this field into the target frame.

Fourth, decide per patch whether to inject the warped signal. Latent patches are nodes on a triangulated grid. For each triangle of neighbouring patches we compute the signed-area ratio of its warped image; a flip or a large stretch marks a frustrated plaquette. The frustration score $f_i$ of patch $i$ combines three things: the fraction of its triangles that are frustrated, the cycle inconsistency of source→target→source correspondence, and the disagreement between the warped residual and the model's own target-conditioned velocity over the first few steps. We then choose a lock/release label $s_i \in \{-1,+1\}$ by minimising

$$E(s) = \sum_i (f_i - \tau)\, s_i \;-\; J \sum_{\langle ij \rangle} s_i s_j ,$$

where $\tau$ is a threshold, $J>0$ couples neighbouring patches $\langle ij\rangle$ so that decisions come in contiguous regions (this is how we address the seam bleed described by Zheng et al.), and $s_i=+1$ means release. With $J>0$ this energy is a ferromagnetic Ising model, which a graph cut solves exactly in milliseconds. Released patches are denoised freely under the target pose condition; locked patches receive the warped residuals and values. The decision costs about three extra network evaluations for the velocity probe, which we will report.

We will also analyse where the frustration score fires relative to SMPL-X visibility. If it mostly reproduces the visibility mask, the physics-derived component adds nothing, and we will say so.

Baselines

The naive reference is plain Euler inversion followed by regeneration under the target-pose condition. This measures how much fidelity is lost without any correction.

The strongest existing competitor is DirectEdit run unchanged with the target-pose condition swapped in. It is the best reported inversion editor on FLUX, and its paper names pose change as its failure mode, so this is the direct test of our claim. The only reported numbers on Lacuna are for PIE-Bench semantic edits (background PSNR 32.63 dB, SSIM 93.49%). For pose edits we found no reported result on Lacuna for DirectEdit on 4D-Dress or any paired pose benchmark, so we will measure it ourselves. FreeInv under the same protocol is a second inversion baseline (reported 29.24 dB FLUX reconstruction PSNR on PIE-Bench).

As the strongest pose-driven method without inversion, we run GenFusion from a single source frame. Its reported 4D-Dress figures come from a monocular-video setting and are not directly comparable, so we rerun it. Null-text inversion followed by pose-conditioned regeneration represents per-image optimisation; FreeInv reports it at 148 s per image, against about 4 s for training-free methods.

Two ablations isolate the key ingredient. The first warps residuals with injection always on. The second warps residuals with release decided purely by SMPL-X visibility and disocclusion. All methods share the backbone, the pose adapter, the SMPL-X fits, 60 function evaluations as in DirectEdit, a fixed guidance scale, and an equal hyperparameter sweep of 20 configurations tuned on a held-out split.

Beating DirectEdit means lower LPIPS against the ground-truth target frame and higher background PSNR, with 95% bootstrap confidence intervals that do not overlap. It also means staying within 1 dB of DirectEdit's PIE-Bench reconstruction when the pose is unchanged.

Why not just…?

Why not just use DirectEdit with a new pose condition and a looser mask? Loosening the mask discards the residuals over the whole body, and the body is exactly where identity and texture live. The file's evidence that DirectEdit's advantage comes from residual replay is the 32.63 versus 25.20 dB gap. Our first baseline measures whether a masked DirectEdit is already good enough. If it matches us, the transport idea is unnecessary.

Why not just skip inversion and render the new pose feed-forward, as GenFusion does? Feed-forward renderers regenerate appearance from learned priors. From one frame they cannot reproduce fine texture or an unseen background exactly, which is the fidelity inversion exists to provide. That said, GenFusion is a real threat on the metrics that reward plausibility, such as FID and FVD. We report pixel-level fidelity and plausibility separately, so that neither side can win on the metric that favours it by construction.

Why not just warp by SMPL-X visibility instead of computing frustration? This is the most serious alternative, and ablation two is designed to test it. Visibility is correct wherever the mesh matches the body. It is wrong on dresses, coats, and hair, where the mesh is not the surface. The frustration score uses the warped residuals and the model's velocity, so it can detect conflict off the mesh. If the two tie on the 4D-Dress loose-clothing subset, we drop the frustration component and publish the simpler rule.

Why not just optimise the latent per image? Per-image optimisation is 37× slower according to FreeInv's timings. Han Zhang et al. also suggest that optimisation does not remove the ODE's inherent instability. We include null-text inversion so the fidelity claim is tested empirically rather than assumed.

Proposed Experiments

The main benchmark is built from 4D-Dress, which GenFusion uses and which has static cameras. Two frames of the same subject in different poses form a source image, a target SMPL-X, and a ground-truth target image; the background matches because the camera does not move. We stratify by joint-angle change and by clothing looseness. THuman2.1 and MVHumanNet renders provide a second paired set with a controlled viewpoint.

Metrics on the target frame are PSNR, SSIM, and LPIPS. Background PSNR and SSIM are computed outside the union of the source and target masks, reusing DirectEdit's protocol. We also report an identity similarity score from a face embedder and the error of a 3D pose re-estimated from the output against the target, which verifies that the edit actually happened.

A sanity run on PIE-Bench with unchanged pose checks that the warping machinery does not degrade DirectEdit's reconstruction fidelity. TikTok in-the-wild clips test robustness to poor SMPL-X fits. On these clips we also inject deliberate limb-swap errors to measure how gracefully each method fails, since the file names fitting sensitivity as the known weakness of pose-conditioned systems.

The decisive checks are three. The main claim holds if our method beats DirectEdit-with-target-pose on target LPIPS while keeping background PSNR within 1 dB on large-angle edits. The frustration idea holds if our method beats the visibility-mask ablation on the loose-clothing subset. The residual transport, as opposed to noise transport alone, is justified if it outperforms warping only the initial noise, which the instability result predicts should fail. Failing the second check would leave a useful but simpler method. Failing the first would show that pose-aware residual transport does not solve the rigidity problem.