Real-Time Autoregressive Generation for Interactive Human Motion

44 concepts39 papers3 proposals

The Latency Challenge in Interactive Motion

Traditional generative models for human motion typically operate in a "space-time" batch mode. They accept a complete text prompt or an entire audio track and synthesize a full, global motion sequence all at once. While this approach yields high-fidelity results, it introduces a fundamental architectural flaw for interactive applications like video games, live streaming, or social virtual reality: it is non-causal. To generate the very first frame of movement, the model must have access to the future context of the driving signal.

This creates an unacceptable latency bottleneck. Interactive systems require an online processing mode where characters respond immediately to a continuous, unpredictable stream of user inputs or environmental changes. The challenge lies in transitioning generative models from offline batch processors to low-latency, frame-by-frame synthesizers without sacrificing physical realism or temporal consistency.

Causal and Autoregressive Formulations

To achieve real-time responsiveness, recent architectures shift from global sequence generation to causal, frame-by-frame synthesis. In an autoregressive setup, the model predicts a narrow window of future frames conditioned strictly on a historical window of past states and immediate external control signals.

[Past Frames] + [Current Input (Audio/Text)] ---> [Autoregressive Model] ---> [Next Frame]
       ^                                                                           |
       |___________________________________________________________________________|

For instance, Jingbo Wang et al. developed an autoregressive motion diffusion model in Interactive Character Control with Auto-Regressive Motion Diffusion Models (A-MDM). By predicting each frame based on the previous one, the authors enabled frame-by-frame generation that allows for immediate interactive control, overcoming the heavy computational constraints of traditional space-time diffusion models.

This causal constraint is also vital when synthesizing human-to-human interactions. In ReGenNet: Towards Human Action-Reaction Synthesis, Liang Xu et al. treated reaction generation as an asymmetric, online process. By restricting the model from accessing future actor frames, they mimicked human reaction speeds, producing highly plausible interactive behaviors suitable for live environments like augmented reality.

For streaming audio-driven animation, Dingcheng Zhen et al. introduced a similar sequential approach in Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation. Their framework utilizes an autoregressive transformer to predict motion tokens sequentially based on streaming audio features, avoiding the multi-step iterative denoising processes typical of standard diffusion to achieve low-latency execution.

Accelerating the Denoising Pipeline

While autoregressive modeling solves the causality problem, integrating diffusion models still introduces a heavy computational burden due to their iterative denoising steps. To achieve true real-time performance (exceeding 30 to 60 frames per second), researchers must drastically compress or bypass the traditional denoising pipeline.

One powerful approach is latent consistency distillation. In MotionLCM: Real-Time Controllable Motion Generation via Latent Consistency Model, Wenxun Dai et al. demonstrated that complex latent diffusion models can be distilled into a consistency model. This mathematical formulation maps any point along the diffusion trajectory back to the origin ($t=0$) in a single inference step. This reduces computation time to approximately 30 milliseconds, enabling high-fidelity, text-conditioned motion generation that is orders of magnitude faster than traditional iterative denoising.

Other frameworks accelerate inference by combining diffusion with adversarial training or few-step distillation:

Bypassing Denoising via Masking and State Space Models

An alternative strategy for eliminating diffusion latency is to abandon iterative denoising entirely, replacing it with high-throughput sequence modeling architectures like masked transformers or state space models.

chuan guo et al. explored this in MoMask: Generative Masked Modeling of 3D Human Motions by combining hierarchical Residual Vector Quantization (RVQ) with bidirectional masked transformers. Instead of gradually removing noise, the model performs iterative masked token prediction. This mechanism requires significantly fewer inference steps than diffusion while maintaining competitive performance and statistical diversity. Pinxin Liu et al. leveraged a similar masking-based transformer architecture in Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representation, achieving near real-time synthesis of long-sequence videos by exploiting the chronological alignment between audio and motion.

Diffusion:       [Noise] -> [Step T] -> ... -> [Step 1] -> [Motion]  (Slow, Iterative)
Masked Token:    [Masked Tokens] -> [Predict Unmasked] -> [Motion]   (Fast, Few Steps)
State Space:     [Input] -> [Linear State Update] -> [Motion]        (Ultra-Fast, O(1) Inference)

Taking efficiency further, Haonan Han et al. introduced MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models. By replacing computationally heavy transformers and diffusion models with Selective State Space Models (SSMs), the authors decoupled sequence length from computational complexity during inference. This architecture achieves generation speeds up to 33 times faster than diffusion-based models, offering a highly efficient path for real-time interactive environments.

Combating Error Accumulation and Drift

A notorious drawback of autoregressive generation is error accumulation. Because the model feeds its own past predictions back into itself as input, small inaccuracies at frame $t$ compound over time. Left unchecked, this causes severe artifacts like "foot sliding," joint distortion, or catastrophic drift where the character deforms entirely.

To stabilize long-term generation, models must enforce structural, geometric, or kinematic constraints. In Human Motion Diffusion Model, Guy Tevet demonstrated that predicting the actual motion sample itself rather than the noise at each diffusion step allows for the direct integration of geometric losses, such as explicit foot contact constraints, which effectively grounds the character and prevents sliding.

Another solution involves rethinking the underlying coordinate system. In Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints, Jiaxu Zhang et al. abandoned traditional hierarchical joint modeling in favor of predicting global rotations directly. In a standard local kinematic chain, an error in the shoulder joint propagates down the arm, magnifying exponentially at the fingertips. By decoupling joints from their kinematic ancestors, their framework eliminates this cumulative error propagation, significantly stabilizing the generation of fine-grained hand and feet movements.

Finally, quantizing the motion space into discrete tokens bounds error propagation. As shown by Alexander Richard et al. in MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement, a categorical latent space allows an autoregressive model to sample plausible, temporally consistent sequences even when driving modalities are missing or noisy. The discrete codebook vectors act as a natural buffer: because sub-threshold errors are snapped to the nearest valid token, continuous drift is prevented from destabilizing the animation over long horizons.

Key Foundational Work

Deep Video-Based Performance Cloning (2019) — contributed 1 concept elements to this direction.
Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_real_time_motion_generation_78569bef,
  title = {real-time motion generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-motion-generation/rdr_78569bef61c143f983e3cf00dd9cc199},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_real_time_motion_generation_78569bef_v1,
  title = {real-time motion generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-motion-generation/rdr_78569bef61c143f983e3cf00dd9cc199/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake