Root-Relative Reference Attention for Character Animation Across Large Displacements

Systems that animate a person from a single photo, such as Animate Anyone, keep fine detail like fabric prints and logos by running a second copy of the image generator over the reference photo, called a ReferenceNet. The generator then looks up the photo's features at every step through attention, where each output location gathers features from the photo locations it is most similar to. This works well on studio benchmarks, where the person stays roughly in place. It is less clear that it holds when the person walks across the frame, shrinks with distance, or is followed by a moving camera. In those cases the photo and the output frame no longer share a coordinate system, and the lookup has to match features between locations that are far apart. We propose to run that lookup in the character's own moving coordinate frame. Every reference and output position is re-expressed relative to the character's root (the pelvis) and divided by its body height. The full-frame canvas is still generated jointly, so the background, shadows, and ground contact remain coherent.

Put simply, global–local motion disentanglement means splitting motion into where the body goes and how it moves around its own center. We use that split to decide where the detail lookup happens, rather than as a way to generate trajectories. The claim is testable. Detail fidelity in existing ReferenceNet animators should fall as the displacement between the reference pose and the target pose grows. Root-relative attention should flatten that curve without the artifacts of cropping the person and pasting them back into the scene.

Core Claim and What Is New

Work that separates global from local motion has so far used the split for prediction or for steering generation. Disentangling Human Dynamics forecasts a neck-joint path and neck-relative joints separately. GLASS estimates 2D shifts and codes local actions as discrete tokens, but decodes with a recurrent network that blurs texture. TC4D moves a NeRF bounding box along a spline and deforms it locally, with no image-space reference injection at all. MoCa is the closest system that pairs a ReferenceNet with motion disentanglement. However, its split separates camera motion from object motion using wavelet frequency bands, and its own stated limitation is that it cannot place or steer the foreground object.

The detail-preservation line works on the opposite side. Animate Anyone, MagicAnimate, Champ and Animate Anyone 2 inject dense reference features, but always in absolute canvas coordinates, driven frame by frame by poses that have not been decomposed. None of these papers changes the coordinate system of the reference lookup itself. That change is the contribution.

Concretely, let $r_t$ be the 2D root position and $s_t$ the projected body height in output frame $t$, and let $r_0, s_0$ be the same quantities in the reference photo. Both come from the driving pose and reference pose that these systems already take as input. A token at pixel position $p$ in frame $t$ receives the root-relative coordinate

$$\tilde p = \frac{p - r_t}{s_t},$$

and reference tokens are mapped the same way using $r_0, s_0$. In the attention layers that mix reference and output tokens, we add a rotary positional encoding computed from $\tilde p$ rather than from $p$, together with a soft locality bias $-\lambda\,\lVert \tilde p_{\text{out}} - \tilde p_{\text{ref}} \rVert^2$. The bias pushes each output token to look first at the reference region that occupies the same body-relative place. Here $\lambda$ is a learned per-layer scalar initialized at zero, so training starts from ordinary attention. Self-attention among the output tokens keeps absolute coordinates, which leaves the background and the camera geometry untouched. Because the global path $\{r_t, s_t\}$ now enters only through this coordinate change, the user can edit it independently of the local pose sequence. This gives explicit placement of the character on the canvas, which MoCa reports it cannot do.

Research Program

The first stage is diagnostic, and it determines whether the project should continue. The Stable-Diffusion UNet behind these animators has no explicit positional encoding in its attention; position enters only implicitly through the convolutions. The claimed failure under large displacement, in which ReferenceNet tears or drifts texture, is plausible but has not been measured anywhere in the material we have. We will build a displacement-controlled test set. Each UBC Fashion test video is placed on a larger canvas and moved along known trajectories, with known scale changes and synthetic camera pans. Detail fidelity is then measured as a function of the distance between reference and target in $(r, s)$. If public Animate Anyone-style models show a flat curve, the premise is false and we report that as a negative result.

The second stage builds the model. We start from a reproduced Animate Anyone setup: an SD-1.5 denoiser, a ReferenceNet, a pose guider, and temporal layers. We replace the positional treatment of the reference–output attention with the root-relative encoding and the locality bias, and train on the same UBC Fashion and TikTok data. Training also uses random global translation and scale of the target canvas. This augmentation is necessary because otherwise the training set never contains large displacements. To keep the comparison honest, the baselines receive the same augmentation.

The third stage looks at the coupling between global and local motion. Once the path can be edited, we change walking speed along a path while keeping the local pose sequence fixed. We then measure foot sliding in the generated video by tracking foot keypoints that should be stationary during stance. TC4D documents the same artifact when the global and local components are not synchronized. We will test whether anchoring $r_t$ to the stance foot reduces it, as opposed to treating the pelvis path as free. This stage is secondary, and the project's success does not depend on it.

Baselines

The naive reference point is an unmodified Animate Anyone reproduction trained on the same data without translation augmentation. It shows how far current practice degrades across displacement bins. On standard UBC Fashion, Animate Anyone reports an FVD of 81.6, against 175.4 for DisCo (Animate Anyone). Our reproduction must land near 81.6 on the untranslated split before any comparison counts.

The strongest existing method is Champ (Champ), which adds rendered SMPL depth, normal, and semantic maps alongside a ReferenceNet. It is the most serious competitor because its 3D body-model guidance may already supply enough geometric correspondence to survive displacement. No reported result was found on Lacuna for Champ's numerical FVD, PSNR or SSIM on UBC Fashion; the file says only that it outperforms MagicAnimate and Animate Anyone. We will therefore rerun it ourselves on the same data and the same displacement-augmented training, under the same number of GPU-hours and the same hyperparameter search budget. Beating Champ means equal or better person-region LPIPS and FVD in the smallest displacement bin, together with a significantly flatter degradation slope across bins, with confidence intervals taken over test subjects.

The crop-and-paste pipeline is a separate baseline. It runs Animate Anyone on a root-centered, scale-normalized crop and then composites the crop into a background generated or inpainted along the path. This is the obvious engineering version of the same idea.

Two ablations isolate the key ingredient. The first trains with translation augmentation alone, keeping absolute coordinates. The second uses the root-relative encoding without the locality bias. If the first ablation matches the full model, the mechanism adds nothing.

For context on detail metrics, DreamPose reports UBC Fashion LPIPS of 0.068 after per-subject decoder finetuning (DreamPose). Because that number depends on a roughly 30-minute optimization per subject, it is not a like-for-like comparison.

Why not just…?

Why not just crop the person, animate the crop, and paste it back? This is the most serious threat, because it achieves canonicalization for free in the large-displacement bins. It cannot generate shadows, ground contact, occlusion by scene objects, or lighting that changes along the path. It also introduces seams at the crop border. Animate Anyone 2 found that tying generation tightly to segmentation contours produces edge artifacts, and it had to deliberately loosen mask correspondence (Animate Anyone 2). We will settle the question with metrics on the full frame and on a boundary band, plus a forced-choice user study on composite realism. If crop-and-paste matches us on all three, we will drop the architectural change and report the pipeline result instead.

Why not just augment training with random translations and let the network learn invariance? GLASS argues that without an explicit split a model must learn each action at every screen position, which is sample-inefficient (GLASS). In forecasting, removing the explicit split worsened keypoint displacement error on JAAD from 10.9 to 18.2 (Disentangling Human Dynamics). Neither result concerns reference attention, though. The augmentation-only ablation answers the question directly, and we will also report how each variant behaves as the training data is reduced to 25% and 50%. That is where a sample-efficiency advantage should appear.

Why not just use 3D body-model guidance as Champ does? Body-model maps tell the generator where body parts are in each frame. They do not change how reference features are retrieved, so texture lookup may still drift across large shifts. Champ also inherits monocular fitting errors and the cost of rendering every frame. If Champ's degradation curve is already flat on the displacement benchmark, our contribution shrinks to cheaper conditioning. We will say so, and we will test whether root-relative attention adds anything when placed on top of Champ.

Proposed Experiments

The decisive experiment is the displacement-binned degradation curve on translated UBC Fashion and translated TikTok. We measure person-region LPIPS, PSNR and SSIM against ground-truth frames shifted the same way, together with FVD per bin, for every baseline and ablation. The mechanism is supported only if our slope is significantly flatter than both the augmentation-only ablation and Champ, while performance in the small-displacement bin stays the same.

For real rather than synthetic displacement, we use the Tennis clips from GLASS, which contain genuine court-wide motion and camera movement. We report FVD and a user study there, since no per-frame ground truth exists for edited paths.

The trajectory-editing test retimes and reroutes the root path on held-out TikTok and Tennis poses. We measure placement error between the requested $r_t$ and the detected root in the output, along with stance-foot sliding with and without foot anchoring.

Cost is reported as memory and seconds per frame relative to the Animate Anyone reproduction. The added positional terms should leave both essentially unchanged.