When Can a Real-Time Avatar Ignore Its Effect on You? Measuring and Modeling Back-Action in Interactive Motion Generation

Real-time systems that animate a virtual partner (an avatar in VR, a game character, or a humanoid robot dancing or shaking hands with a person) usually treat the human as a one-way input. The human moves, and the avatar reacts. The human's future motion is assumed not to depend on what the avatar does. Physicists have a name for this kind of shortcut. In the weak-coupling approximation, an environment is treated as a passive, memoryless reservoir that absorbs a system's influence without pushing back. That approximation fails in a predictable way once the interaction becomes as strong as the system's own internal dynamics. At that point the environment reorganizes around the system's trajectory, and models that ignore this push-back can only fit the data by inflating their noise. This note proposes to carry that diagnosis, and a cheap correction for it, into interactive motion generation. The goal is not a loose metaphor. It is a measurable quantity, a predicted failure mode, and a constant-cost fix.

The concrete claim has two parts. Each interaction can be assigned a coupling strength, estimated from motion-capture data, that measures how much one person's motion is driven by the other's. One-way reactive generators should break down specifically where this coupling is high, for example pulling, partner dancing, or catching someone, and should be adequate where it is low, for example waving or pointing. Where coupling is high, the avatar needs a small internal model of how the human will respond to the avatar's own next move. That response model can be carried as a handful of extra recurrent state variables, so it adds essentially no per-frame latency.

Core Claim

Current online interaction generators all adopt the weak-coupling approximation in the same structural way. ReGenNet (ReGenNet: Towards Human Action-Reaction Synthesis) generates a reactor with a causal mask, so the reactor sees the actor only up to time $t$. The actor is never modeled as responding to the reactor. Human-X (Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis) pairs an autoregressive diffusion planner with a physics tracking controller. Its authors state that it cannot handle shared-goal collaborative tasks. Both systems work well on the benchmarks they report. The proposal argues that their remaining errors are concentrated where the actor's motion depends on the reactor's, and that this dependence can be quantified before any generator is trained.

The physics literature supplies three pieces of support, each playing a different role. The first is motivation. Trappen et al. (Dissipative Landau-Zener tunneling in the crossover regime from weak to strong environment coupling) study a superconducting qubit swept through an energy-level crossing while coupled to its environment. A weak-coupling description fits the sweep data only if the assumed noise is about eight times larger than the noise measured in static experiments.

The motion analogue is a falsifiable prediction. A one-way reactor trained on dyadic (two-person) data should learn inflated predictive variance in high-coupling segments, because it cannot represent the partner's response and absorbs it as randomness. The same compensation through noise may explain why one-way models look diverse yet miss contact.

The second piece supplies the method. Li (Markovian Embedding Procedures for Non-Markovian Stochastic Schrödinger Equations) shows how a response that depends on the whole past can be written as a sum of exponentials. Each exponential is carried as one auxiliary variable that updates locally in time. This turns a computation whose cost grows with history length into a constant-cost update per step. Li also shows that the fit must capture the slow, low-frequency part of the response, or long-horizon quantities drift.

The third piece is a warning. Lin et al. (Time-optimal single-scalar control on a qubit of unitary dynamics) show that controllers which discard fast components, as the rotating-wave approximation does, are about 20% slower than the optimum. If the response model is fitted only to slow components, it will miss fast reflexes such as catching or bracing.

Research Program

The first component is a coupling-strength measure. For a dyad with poses $x_t$ (the avatar role) and $y_t$ (the human role), fit a causal linear-response model of the human's future motion:

$$\ddot y_t = \sum_{s<t} H(t-s)\, y_s + \sum_{s<t} K(t-s)\, x_s + \epsilon_t .$$

Here $H$ is the human's self-dynamics kernel, $K$ is the cross kernel describing how the human responds to the avatar, and $\epsilon_t$ is the residual. Both kernels are fitted in a learned low-dimensional pose space rather than raw joints.

The coupling strength $g$ of a segment is the fraction of the human's predictable acceleration variance that is explained only by the cross term. It is computed on held-out frames so that it is not inflated by overfitting. This is a Granger-style quantity, and it plays the role of the coupling-to-internal-scale ratio in physics. We will compute $g$ per category and per segment on Inter-X (Inter-X: Towards Versatile Human-Human Interaction Analysis), whose actor-reactor causal annotations give an independent check: categories annotated as clearly one-directional should have low $g$ in the reactor-to-actor direction. We will also compute $g$ on InterHuman and NTU120-AS.

The second component is a back-action-aware causal reactor. The base generator is a causal reactor in the style of ReGenNet, using a few-step sampler so it runs at 30 frames per second. To this base we add $M$ auxiliary states $z^{(k)}_t$, with $M$ between 3 and 7, matching the pole counts Li found sufficient. Each state obeys $z^{(k)}_{t+1} = e^{-\gamma_k} z^{(k)}_t + B_k x_t$, where $\gamma_k$ is a learned decay rate and $B_k$ a learned input map. The predicted human response is then $\hat{\delta y}_t = \sum_k C_k z^{(k)}_t$, with $C_k$ a learned output map.

This is the sum-of-exponentials embedding of the cross kernel $K$. At each frame the generator takes a candidate next pose $x_{t+1}$, updates the auxiliary states, and conditions on the predicted human response. This gives a one-step lookahead in the avatar's own action at the cost of a few vector updates. Following Li, one exponential with a very slow decay is reserved, so that slow drift such as the partner gradually yielding ground is represented. The decay rates of the remaining exponentials are left free to go fast, following Lin et al.'s warning about discarding fast components.

The third component is closed-loop evaluation. Offline datasets cannot show how a real human would react to a generated avatar. We will therefore train a separate partner simulator with a different architecture (a joint causal transformer) on held-out subjects. During evaluation the simulator plays the human, reacting to the avatar's generated motion. Because the simulator is not the response model inside the reactor, the reactor cannot simply exploit its own predictions. A small user study in VR will check whether simulator-based rankings agree with human judgments.

Baselines

The naive reference point is a one-way causal reactor with no auxiliary states: the same backbone, data, and sampler as ours. This is the weak-coupling approximation implemented exactly. Comparing against it isolates whether modeling the partner's response helps at all.

The strongest existing published systems are ReGenNet and Human-X, and they are reported on different benchmarks. On NTU120-AS, ReGenNet reports FID 0.90 against 5.31 for an MDM-GRU baseline. On Inter-X at 30 frames per second, Human-X reports FID 0.975, penetration 0.118, skating 0.092, and floating 8.650. The Inter-X benchmark's own reaction-generation baseline (AGRoL) reports FID 4.386. For InterHuman, no reported result was found on Lacuna.

We will retrain ReGenNet and the Human-X planner on our splits with the same tuning budget. Each system will also be run with and without the Human-X physics controller, which is complementary to our change rather than a substitute for it. Beating these systems means two things. On aggregate FID and penetration on Inter-X, we must match within noise. On the top quartile of segments by coupling $g$, we must clearly win on contact-timing error and closed-loop task success. A win only on low-$g$ segments would count against our explanation.

The key ablation replaces the exponential cross-kernel states with an equal number of extra parameters that see the same partner history but cannot be driven by the avatar's own candidate pose. This tests whether lookahead in the avatar's own action, rather than added capacity or memory, is what matters. A second ablation removes the slow exponential to test Li's low-frequency argument directly.

Why not just…?

Why not just train a single joint causal model of both people and roll the partner forward in imagination? This is the most serious threat. A joint model trained on dyadic data implicitly contains back-action. If the avatar imagines several partner futures per frame, it obtains anticipation without any explicit kernel. Our answer is partly about cost, since imagined rollouts multiply the per-frame forward passes and threaten 30 frames per second. It is also an empirical question we will settle directly. We will include joint-rollout anticipation, given the same compute per frame, as a baseline. If it matches our method at equal latency, the method contribution shrinks to the coupling diagnostic, and we will say so.

Why not just give the reactor a longer memory of the partner, for example with a state space model? MambaTalk (MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models) gets long history at linear cost, reporting 0.019 seconds of latency per second of gesture on BEAT2. Our auxiliary states are themselves a small linear recurrence, so the distinction is not the recurrence. The distinction is what drives it: the avatar's own candidate action, used to predict the partner's response. A long one-way memory still assumes the partner ignores the avatar. The capacity-matched ablation above tests exactly this point.

Why not just add a physics tracking controller, as Human-X does? The controller repairs penetration and skating after the kinematic planner has committed to a motion. It does not anticipate that a pull will move the partner. We expect the controller and our response states to stack. We will report both combinations, and we predict that the controller alone helps on low-$g$ segments but not on contact timing in high-$g$ segments.

Proposed Experiments

The first decisive check needs no generator at all. If coupling $g$ does not separate Inter-X categories in a way consistent with the actor-reactor annotations, the premise fails.

The second check tests the noise-compensation prediction. We will train the one-way reactor and measure its predictive variance as a function of $g$. The prediction is that variance rises with $g$ while contact accuracy falls, mirroring the inflated-noise signature of Trappen et al. The back-action-aware reactor should flatten that curve.

The third check plots the error gap between our reactor and each baseline against $g$ binned into quartiles, using FID, MMDist, contact-timing error, and closed-loop task success with the held-out partner simulator. The claim is supported only if the gap is near zero at low $g$ and grows with $g$.

Finally, we will measure per-frame latency on fixed hardware against the 30 Hz budget that Human-X operates under. We will also test fast-reflex segments separately to check whether the fitted decay rates actually reach the fast components Lin et al. warn are easily lost.