Learning Human Motion with Temporally Conditional Mamba

Quang Nguyen, Tri Le, Baoru Huang, Minh Nhat Vu, Ngan Le, Thieu Vo, Anh Nguyen

Introduction

Generating realistic human motion that remains synchronized with time-varying external signals—such as the rhythmic beat of music or the shifting perspective of egocentric video—remains a significant challenge in computer vision. While Diffusion-based frameworks paired with Transformers have become the standard, they often struggle with fine-grained temporal precision. This paper introduces Temporally Conditional Mamba (TCM), a novel architecture designed to improve synchronization by injecting conditioning data directly into the recurrent dynamics of a Selective State Space Model.

The Temporal Alignment Challenge

Most contemporary motion generation models rely on Cross-Attention mechanisms to integrate conditioning signals. While effective at capturing global correlations across a sequence, cross-attention is inherently non-causal and treats the sequence as a holistic set of tokens. This "global" view often results in a lack of precise step-by-step alignment. For instance, in dance generation, a model might produce movements that match the general genre of the music but fail to synchronize specific footfalls with the actual beat of the track.

The authors argue that human motion is inherently a continuous, state-dependent process. To achieve high-fidelity synchronization, the conditioning signal must act as a continuous influence on the motion stream, dictating the "flow" of movement in real-time rather than as a secondary lookup table.

Methodology

The proposed framework utilizes a diffusion-based approach, replacing standard Transformer blocks with a specialized dual-Mamba architecture consisting of two primary components:

1. Temporally Conditional Mamba (TCM)

TCM is the core innovation of this research. In a standard Mamba block, the system matrices ($B$ and $C$) govern how the internal hidden state evolves over time. TCM modifies these matrices to be dynamically dependent on the external condition. By employing an element-wise affine modulation, the input signal (e.g., music embeddings) directly "steers" the recurrent dynamics of the motion generation at every frame. This ensures that the model's internal transition logic is inherently aware of the temporal context.

2. Spatial Mamba

While TCM focuses on temporal evolution, Spatial Mamba addresses the relationships between body parts. By rearranging data to treat joints as a sequence, the model learns complex spatial dependencies, such as the coordinated movement between a shoulder's rotation and a wrist's trajectory.

3. Global Coherence

To maintain overall sequence stability, the model incorporates Adaptive Layer Norm (AdaLN). This component scales and shifts features based on the global timestep and condition embedding, ensuring that the generated motion remains coherent across the entire duration of the sequence.

Experimental Evaluation

The model was evaluated across four diverse motion tasks using several benchmark datasets:

  • AIST++: Dance motion paired with music.
  • ARES: Egocentric video paired with indoor human motion.
  • EgoExo4D: Multimodal estimation combining video and music.
  • Omomo: Human-object interaction and manipulation.

Key Results

  • Superior Alignment: On the AIST++ benchmark, TCM achieved a Beat Alignment Score (BAS) of 0.2761, significantly outperforming both the Ground Truth (0.2374) and the Cross-Attention baseline (0.2411). This indicates that the model generates motion that is more rhythmically consistent than the original human recordings in some metrics.
  • Enhanced Realism: TCM consistently yielded lower FID scores compared to Vanilla Mamba and Cross-Attention models. These gains were particularly pronounced as sequence lengths increased to 25 seconds, demonstrating superior long-range stability.
  • Computational Efficiency: Despite its performance, TCM remains efficient. With approximately 26.8M parameters, it is significantly leaner than comparable Cross-Attention Transformer architectures (~41.3M) while maintaining similar inference speeds to standard Mamba models.

Discussion and Insights

The research highlights that for temporal alignment, recurrence is a feature, not a bug. While the industry has largely pivoted toward attention mechanisms for their global reach, Mamba’s recurrent nature is biologically and physically better suited to modeling continuous motion.

Visualization of the TCM block's learned parameters via t-SNE revealed that the model naturally clusters features by music genre (e.g., Western Pop vs. Jazz). This suggests that the model is not merely memorizing motion patterns but is adapting its internal "physics" to the specific style of the input signal.

Limitations and Future Work

Despite its strengths, TCM is specifically optimized for temporal signals. It may not offer the same advantages for static inputs, such as a single text prompt, where there is no time-varying signal to track. Additionally, the model can struggle with extremely abrupt, jerky movements in "ego-to-motion" tasks, leading to occasional lag. Finally, like many current generative models, TCM can produce "foot-skating" artifacts. Future work will likely involve integrating explicit kinematic constraints or foot-contact loss functions to further enhance physical realism.

Summary updated

Abstract

Learning human motion based on a time-dependent input signal presents a challenging yet impactful task with various applications. The goal of this task is to generate or estimate human movement that consistently reflects the temporal patterns of conditioning inputs. Existing methods typically rely on cross-attention mechanisms to fuse the condition with motion. However, this approach primarily captures global interactions and struggles to maintain step-by-step temporal alignment. To address this limitation, we introduce Temporally Conditional Mamba, a new mamba-based model for human motion generation. Our approach integrates conditional information into the recurrent dynamics of the Mamba block, enabling better temporally aligned motion. To validate the effectiveness of our method, we evaluate it on a variety of human motion tasks. Extensive experiments demonstrate that our model significantly improves temporal alignment, motion realism, and condition consistency over state-of-the-art approaches. Our project page is available at https://zquang2202.github.io/TCM.
BibTeX
@inproceedings{nguyen2025learning,
  author = {Quang Nguyen and Tri Le and Baoru Huang and Minh Nhat Vu and Ngan Le and Thieu Vo and Anh Nguyen},
  title = {{Learning Human Motion with Temporally Conditional Mamba}},
  booktitle = {SIGGRAPH Asia},
  year = {2025},
}
Figure