TL;DR
This paper introduces MotionFix, a novel dataset and a conditional diffusion model (TMED) for text-driven 3D human motion editing. MotionFix enables training models to modify existing motion sequences based on natural language instructions, a task previously hindered by a lack of suitable data. TMED demonstrates superior performance in accurately editing motions according to textual descriptions compared to existing methods.
What Problem Does This Paper Address?
Editing 3D human motion based on textual descriptions is a challenging task. While text-to-motion generation has seen progress, fine-grained editing of existing motions presents unique hurdles. These include the scarcity of appropriately annotated training data (source motion, target motion, and edit description) and the difficulty in creating models that can faithfully modify a given motion while adhering to the textual instruction. Existing methods often rely on local modifications or are limited to specific types of edits, failing to capture the broad range of natural language instructions.
Methods
The core of this work lies in two main contributions: a new dataset and a novel model.
- MotionFix Dataset: The authors propose a semi-automatic methodology to create a dataset of motion editing triplets. This involves:
- Candidate Pair Generation: Using a pre-trained text-to-motion retrieval model (TMR) to find pairs of motions from existing motion capture datasets that are semantically similar but have subtle differences.
-
Manual Annotation: Human annotators then provide textual descriptions of the differences between these motion pairs. This approach ensures that the edits are meaningful and can be described concisely.
-
TMED (Text-Driven Motion Editing Diffusion Model): This is a conditional diffusion model designed for motion editing.
- Architecture: It builds upon diffusion models, similar to InstructPix2Pix for image editing.
- Conditioning: TMED is conditioned on three inputs: the source motion, the textual edit instruction, and a noise vector for stochasticity.
- Training: The model is trained to denoise a noisy version of the target motion, guided by both the source motion and the text instruction.
- Classifier-Free Guidance: The model employs classifier-free guidance to balance the influence of the source motion and the text instruction during generation.
Datasets
- MotionFix (New Dataset): This is the primary dataset introduced in the paper. It contains 6730 triplets of (source motion, target motion, edit text). It's designed specifically for training and evaluating text-driven motion editing.
- AMASS: Used as the source for motion data to create the MotionFix dataset.
- HumanML3D: Used for training baseline models.
Key Insights and Ideas
- The Importance of Triplet Data: The paper strongly emphasizes that training on (source motion, target motion, edit text) triplets is crucial for effective motion editing, outperforming models trained only on text-motion pairs.
- Semi-Automatic Data Curation: The method for generating candidate motion pairs using TMR similarity is a clever way to overcome the bottleneck of manual data creation for motion editing.
- Balancing Source and Text Guidance: The use of separate guidance scales for the source motion and text instruction in TMED is essential for achieving both faithfulness to the original motion and adherence to the edit description.
- Language Ambiguity and Motion Editing: The paper acknowledges that natural language can be ambiguous, and the model's ability to handle this, while still producing plausible edits, is a key aspect.
Key Results
- Quantitative Performance: TMED achieves significantly higher retrieval-based metrics (R@1, R@2, R@3, AvgR) for generated-to-target retrieval compared to strong baselines like MDM, MDMS, MDM-BPS, and MDM-BP. For instance, TMED achieves an R@1 of 62.90, while the best baseline (MDM-BP) reaches 39.10.
- Qualitative Results: TMED demonstrates the ability to perform a wide range of edits, including modifying body parts, changing motion speed, temporal adjustments (e.g., "start crawling earlier"), and spatial changes (e.g., "raise your arm higher").
- Dataset Size Impact: Increasing the training data size for MotionFix leads to significant performance improvements, validating the dataset's utility.
- Perceptual Studies: Human evaluators rated TMED's generated motions as closely following instructions, with a mean score of 3.59 on a 5-point Likert scale, only slightly below the ground truth (3.93) and better than the strongest baseline (3.52). TMED was also preferred in pairwise comparisons 65.8% of the time.
Limitations and Future Work
- Motion Similarity Assumption: The reliance on TMR similarity to find "edit-ready" motion pairs is a starting point, but may not always perfectly capture subtle differences or complex edits.
- Motion Length Constraint: The dataset was primarily collected for motions up to 5 seconds, as longer motions made it harder to find suitable dissimilar pairs.
- Generalization to Complex Edits: TMED can struggle with very detailed or complex edit texts, and sometimes fails to maintain perfect faithfulness to the source motion.
- Iterative Editing: While the model can be used iteratively, this capability was not deeply explored and is left for future work.
- Future Directions: The authors suggest exploring the iterative editing capability and further scaling up the dataset and model.
Summary updated