Introduction
Generating realistic human motion that remains synchronized with time-varying external signals—such as the rhythmic beat of music or the shifting perspective of egocentric video—remains a significant challenge in computer vision. While Diffusion-based frameworks paired with Transformers have become the standard, they often struggle with fine-grained temporal precision. This paper introduces Temporally Conditional Mamba (TCM), a novel architecture designed to improve synchronization by injecting conditioning data directly into the recurrent dynamics of a Selective State Space Model.
The Temporal Alignment Challenge
Most contemporary motion generation models rely on Cross-Attention mechanisms to integrate conditioning signals. While effective at capturing global correlations across a sequence, cross-attention is inherently non-causal and treats the sequence as a holistic set of tokens. This "global" view often results in a lack of precise step-by-step alignment. For instance, in dance generation, a model might produce movements that match the general genre of the music but fail to synchronize specific footfalls with the actual beat of the track.
The authors argue that human motion is inherently a continuous, state-dependent process. To achieve high-fidelity synchronization, the conditioning signal must act as a continuous influence on the motion stream, dictating the "flow" of movement in real-time rather than as a secondary lookup table.
Methodology
The proposed framework utilizes a diffusion-based approach, replacing standard Transformer blocks with a specialized dual-Mamba architecture consisting of two primary components:
1. Temporally Conditional Mamba (TCM)
TCM is the core innovation of this research. In a standard Mamba block, the system matrices ($B$ and $C$) govern how the internal hidden state evolves over time. TCM modifies these matrices to be dynamically dependent on the external condition. By employing an element-wise affine modulation, the input signal (e.g., music embeddings) directly "steers" the recurrent dynamics of the motion generation at every frame. This ensures that the model's internal transition logic is inherently aware of the temporal context.
2. Spatial Mamba
While TCM focuses on temporal evolution, Spatial Mamba addresses the relationships between body parts. By rearranging data to treat joints as a sequence, the model learns complex spatial dependencies, such as the coordinated movement between a shoulder's rotation and a wrist's trajectory.
3. Global Coherence
To maintain overall sequence stability, the model incorporates Adaptive Layer Norm (AdaLN). This component scales and shifts features based on the global timestep and condition embedding, ensuring that the generated motion remains coherent across the entire duration of the sequence.
Experimental Evaluation
The model was evaluated across four diverse motion tasks using several benchmark datasets:
- AIST++: Dance motion paired with music.
- ARES: Egocentric video paired with indoor human motion.
- EgoExo4D: Multimodal estimation combining video and music.
- Omomo: Human-object interaction and manipulation.
Key Results
- Superior Alignment: On the AIST++ benchmark, TCM achieved a Beat Alignment Score (BAS) of 0.2761, significantly outperforming both the Ground Truth (0.2374) and the Cross-Attention baseline (0.2411). This indicates that the model generates motion that is more rhythmically consistent than the original human recordings in some metrics.
- Enhanced Realism: TCM consistently yielded lower FID scores compared to Vanilla Mamba and Cross-Attention models. These gains were particularly pronounced as sequence lengths increased to 25 seconds, demonstrating superior long-range stability.
- Computational Efficiency: Despite its performance, TCM remains efficient. With approximately 26.8M parameters, it is significantly leaner than comparable Cross-Attention Transformer architectures (~41.3M) while maintaining similar inference speeds to standard Mamba models.
Discussion and Insights
The research highlights that for temporal alignment, recurrence is a feature, not a bug. While the industry has largely pivoted toward attention mechanisms for their global reach, Mamba’s recurrent nature is biologically and physically better suited to modeling continuous motion.
Visualization of the TCM block's learned parameters via t-SNE revealed that the model naturally clusters features by music genre (e.g., Western Pop vs. Jazz). This suggests that the model is not merely memorizing motion patterns but is adapting its internal "physics" to the specific style of the input signal.
Limitations and Future Work
Despite its strengths, TCM is specifically optimized for temporal signals. It may not offer the same advantages for static inputs, such as a single text prompt, where there is no time-varying signal to track. Additionally, the model can struggle with extremely abrupt, jerky movements in "ego-to-motion" tasks, leading to occasional lag. Finally, like many current generative models, TCM can produce "foot-skating" artifacts. Future work will likely involve integrating explicit kinematic constraints or foot-contact loss functions to further enhance physical realism.
Summary updated