TL;DR
This paper introduces ATM, a novel framework that significantly improves text-to-motion generation by explicitly identifying and correcting semantic misalignments between textual descriptions and generated human movements. By introducing inter-motion and intra-motion alignment strategies, ATM enhances both the global coherence and local accuracy of generated motions, achieving state-of-the-art results.
What Problem Does This Paper Address?
Generating realistic and semantically accurate human motion from text descriptions is a challenging task. Existing methods often rely on regression-based objectives (like Mean Squared Error) that focus on positional accuracy but fail to capture the nuances of semantic alignment. This leads to two main problems:
- Inter-motion misalignment: Different text descriptions can result in overly similar motions, meaning the model doesn't distinguish between distinct actions.
- Intra-motion misalignment: Within a single motion sequence, desired actions might be missing, inaccurate, or replaced by unintended ones, even if the overall sequence appears plausible.
These issues stem from a lack of explicit semantic supervision, making it difficult to identify what is wrong and how to fix it. The paper aims to bridge this gap by proposing a framework that actively corrects these semantic inconsistencies.
Methods
The core of ATM is a two-pronged approach to tackle semantic misalignment:
- Inter-motion Alignment: This component addresses global sequence-level inconsistencies. It works by:
- Misalignment Mining: Identifying pairs of generated motions that are structurally similar but semantically distinct based on their embeddings in a pre-trained text-motion alignment space.
- Adaptive Supervision: Applying a novel loss function that dynamically adjusts the optimization strength based on the degree of semantic discrepancy. This allows for more flexible correction of diverse semantic relationships, ensuring global text-motion consistency.
- Intra-motion Alignment: This component refines local semantic inaccuracies within motion sequences in an unsupervised manner. It involves:
- Misalignment Mining: Identifying clips within a generated motion that are least aligned with their corresponding ground truth clips.
- Inferred Semantic Proxies: For a misaligned clip, the method retrieves "positive" and "negative" proxy clips from a pool. A positive proxy is semantically similar but currently dissimilar to the target clip, indicating what the target should be. A negative proxy is semantically dissimilar but currently similar, indicating what the target should not be.
- Adaptive Alignment: A triplet loss is used to refine the misaligned clip towards its positive proxy and away from its negative proxy, with an adaptive margin that accounts for the severity of misalignment.
ATM is designed as a plug-and-play module that can be integrated with existing text-to-motion generation models. The overall loss combines the standard MSE loss with the proposed inter-motion and intra-motion alignment losses.
Datasets
The paper uses two standard datasets for training and evaluation:
- HumanML3D: Contains approximately 14,616 motion sequences with 44,970 textual descriptions.
- KIT: Features 3,911 motion sequences paired with 6,278 text descriptions.
No new datasets were introduced in this paper.
Key Insights and Ideas
- Semantic Supervision is Crucial: The paper strongly argues that relying solely on positional regression (like MSE) is insufficient for accurate text-to-motion generation. Explicit semantic supervision is key to addressing misalignment.
- Adaptive Margins Matter: Unlike traditional metric learning losses that apply uniform penalties, ATM's adaptive margins allow for more nuanced handling of semantic relationships, recognizing that not all misalignments are equal.
- Global and Local Alignment are Complementary: The inter-motion alignment tackles overall sequence coherence, while intra-motion alignment refines specific action details. Both are necessary for high-quality generation.
- Unsupervised Local Refinement is Possible: The intra-motion alignment's use of inferred semantic proxies demonstrates an effective way to correct local inaccuracies without requiring explicit, clip-level textual annotations.
Key Results
ATM consistently improves performance across various metrics on both HumanML3D and KIT datasets when integrated with existing models.
- State-of-the-Art Alignment: ATM achieves new state-of-the-art results on text-motion alignment metrics like R-Precision and MModality.
- Improved Generation Quality: When combined with models like MLD, MotionDiffuse, and ReMoDiffuse, ATM significantly enhances their generation performance.
- Enhanced Diversity: The focus on semantic supervision, rather than just positional accuracy, leads to improved diversity in generated motions.
- User Study Validation: A user study confirmed that ATM-enhanced motions exhibit higher action alignment, sequence alignment, and realism compared to other leading methods like MoMask and BAMM.
Limitations and Future Work
The authors acknowledge two primary limitations:
- Differentiability Requirement: ATM's alignment losses require a continuous, differentiable text-motion embedding space. This makes it challenging to directly apply to models that use discrete motion tokens (like VQ-VAE-based models, e.g., MoMask), as the mapping to the codebook is non-differentiable.
- Temporal Alignment Assumption: The intra-motion alignment assumes temporal alignment between generated and real motion clips. Temporal misalignments could affect the accuracy of similarity computations.
Future work will focus on:
- Exploring reparameterization strategies to enable optimization and improve compatibility with discrete motion representations.
- Developing mechanisms to ensure temporal alignment across motion clips.
Summary updated