Introduction
High-quality motion synthesis is a cornerstone of modern digital media, from AAA video games to cinematic visual effects. However, the reliance on massive Motion Capture (MoCap) datasets presents a significant barrier. MoCap data is expensive to produce and often unavailable for specialized needs, such as rare animal movements or unique dance styles.
MotionDreamer addresses this challenge by introducing a "one-to-many" synthesis framework. Unlike traditional models that require thousands of examples, MotionDreamer can learn the underlying "grammar" of a motion from just a single reference sequence, generating a diverse array of high-quality, physically plausible animations.
The Challenge of Single-Instance Synthesis
When standard Generative Masked Transformers are trained on a single motion sequence, they inevitably suffer from overfitting. Because these models typically employ global self-attention, they view the entire sequence as a single entity. With only one training example, the model simply memorizes the global pattern rather than learning the constituent "building blocks" of the movement. This results in "model collapse," where the generator can only replicate the original sequence or produces nonsensical, glitchy artifacts.
The MotionDreamer Framework
MotionDreamer overcomes the overfitting problem by shifting the focus from global sequences to localized patterns. The framework introduces several architectural innovations to ensure both diversity and temporal coherence.
1. Localized Generative Masked Transformer (Local-M)
At the heart of the system is the Local-M Transformer, which utilizes a Sliding Window Local Attention (SlidAttn) layer. By restricting the model's receptive field, it is forced to learn how local motion segments transition into one another. This effectively treats the single reference as a collection of rearrangeable patterns rather than a fixed timeline.
2. Overlap Attention Fusion (AttnFuse)
To prevent the "choppy" transitions that can occur when focusing solely on local windows, the authors developed AttnFuse. This mechanism blends overlapping attention windows, maintaining smooth temporal coherence across the entire generated sequence.
3. Codebook Distribution Regularization
Single-instance learning often leads to VQ-VAE codebook collapse, where the model only utilizes a few entries. MotionDreamer introduces a KL-divergence-based loss ($L_{token}$) to encourage a uniform distribution of codebook entries. This ensures a richer "vocabulary" of motion tokens, capturing fine-grained details that other models might ignore.
4. Differentiable Dequantization
By employing a sparsemax activation function instead of a hard argmax, the model allows gradients to flow more effectively during training. This leads to superior optimization and more natural movement in the final synthesized motion space.
Experimental Results
The authors evaluated MotionDreamer using the SinMotion dataset, which includes 30 human-like sequences (e.g., hip-hop, breakdancing) and 30 animal sequences (e.g., jaguars, spiders). This variety tested the model's ability to handle diverse skeleton topologies and durations.
Quantitative and Qualitative Performance
MotionDreamer was compared against state-of-the-art (SOTA) GAN-based and Diffusion-based methods. Key findings include:
- 19% Improvement in Harmonic Mean: This composite score of faithfulness and diversity significantly outperformed previous SOTA methods.
- Superior Pattern Preservation: In complex motions like a "hip-hop squat-spin," MotionDreamer successfully preserved difficult sub-patterns that caused competing models to lose track of the motion's physics.
- User Preference: In a study with 20 participants, MotionDreamer earned the highest scores for Coverage (faithfulness) and Diversity, while remaining competitive in Naturalness.
Conclusion and Future Work
MotionDreamer demonstrates that "locality breeds diversity." By limiting the transformer's view, the model learns the internal logic of a motion, enabling applications like crowd animation and music-driven synthesis from minimal data. While currently dependent on the quality of the single reference clip, future work may explore few-shot learning (3-5 clips) to further generalize the model's understanding of motion dynamics.
Summary updated