What is the Best Automated Metric for Text to Motion Generation?

Jordan Voas, Yili Wang, Qixing Huang, Raymond Mooney

Abstract

The field of text-to-motion generation has lacked a rigorous evaluation framework that aligns with human perception. Traditional metrics, such as R-Precision and Frechet Inception Distance (FID), are often borrowed from computer vision and fail to capture the nuances of human movement. This paper addresses this gap by establishing a human evaluation benchmark and introducing MoBERT, a multimodal Transformer-based evaluator. MoBERT utilizes joint attention over text tokens and motion chunks to achieve significantly higher correlation with human judgment than existing metrics, providing a new standard for both sample-level and model-level evaluation.

Introduction

Text-to-motion generation—the synthesis of 3D human animations from natural language prompts—has applications ranging from animation and gaming to virtual reality. As generative models like Motion Diffusion Models (MDM) and MotionDiffuse advance, the need for accurate automated evaluation becomes critical. However, the community has largely relied on "borrowed" metrics without verifying their alignment with human ratings. This research systematically evaluates these metrics and proposes a superior alternative that mirrors human assessment of motion naturalness and faithfulness.

The Limitations of Current Metrics

Currently, researchers primarily use R-Precision (measuring retrieval accuracy) and FID (measuring distributional distance). While these metrics are computationally efficient, they suffer from a significant "evaluation gap."

The authors discovered that existing metrics show near-zero correlation with human judgment on a per-sample basis. For example, R-Precision achieved a Pearson correlation of only 0.036 when compared to human ratings. This means that a model could achieve a high R-Precision score while producing animations that humans find unrealistic or unfaithful to the prompt. Furthermore, traditional Coordinate Error (CE) metrics, which measure joint distance, are effective at ranking models based on the "root" joint (pelvis) but fail to distinguish between natural movement and robotic artifacts at the sample level.

Methodology: The MoBERT Framework

To bridge the gap between automated scores and human perception, the researchers developed MoBERT (Multimodal BERT). The framework is built on two primary pillars:

1. Human Evaluation Benchmark

The authors generated 2,000 motions using five state-of-the-art models and prompts from the HumanML3D dataset. They collected 1,400 high-quality human ratings based on two criteria:

  • Naturalness: The realism and fluidness of the motion.
  • Faithfulness: The degree to which the motion matches the textual description.

2. Multimodal Transformer Architecture

Unlike "CLIP-style" models that utilize separate encoders for text and motion, MoBERT employs a shared Transformer Encoder.

  • Motion Chunking: 3D motion is decomposed into overlapping 0.7-second chunks. This captures temporal dynamics more effectively than treating the entire sequence as a single embedding.
  • Joint Attention: By feeding both text tokens and motion chunks into the same Transformer, the model allows for "cross-talk" between modalities. The attention mechanism can simultaneously weigh the relationship between specific words (e.g., "cartwheel") and specific temporal segments of the motion.
  • Training and Loss: The model was trained on a binary classification task to determine if a motion-text pair matches, utilizing a specialized "L2 balanced loss" to mitigate noise in motion descriptions.

Experimental Results

MoBERT demonstrates a substantial improvement over traditional metrics across multiple dimensions:

  • Sample-Level Correlation: MoBERT (specifically the SVR-based version) achieved a Pearson correlation of 0.624 for Faithfulness and 0.528 for Naturalness. In contrast, previous automated metrics typically hovered around a correlation of 0.2.
  • Model-Level Ranking: When used to rank the overall performance of different models, MoBERT achieved a correlation of 0.99 with human rankings. This suggests that MoBERT can serve as a reliable, low-cost proxy for expensive human studies.
  • The "Root" Insight: The study confirmed that while root-joint Coordinate Error is a decent model-level predictor, it lacks the granularity required for sample-level debugging.

Conclusion and Recommendations

The findings suggest a paradigm shift in how text-to-motion models should be evaluated. The authors recommend that researchers move away from a sole reliance on FID and R-Precision. Instead, a holistic evaluation should include:

  1. MoBERT for high-fidelity alignment with human perception.
  2. R-Precision 1-5 for general retrieval performance.
  3. Coordinate Error (Root PV AE) for basic structural accuracy.

Future work will focus on extending MoBERT to full-body meshes beyond skeletons and improving the differentiability of the SVR version to allow for direct optimization during model training.

Summary updated

Abstract

There is growing interest in generating skeleton-based human motions from natural language descriptions. While most efforts have focused on developing better neural architectures for this task, there has been no significant work on determining the proper evaluation metric. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. Since descriptions are compatible with many motions, determining the right metric is critical for evaluating and designing effective generative models. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better. Our findings indicate that none of the metrics currently used for this task show even a moderate correlation with human judgments on a sample level. However, for assessing average model performance, commonly used metrics such as R-Precision and less-used coordinate errors show strong correlations. Additionally, several recently developed metrics are not recommended due to their low correlation compared to alternatives. We also introduce a novel metric based on a multimodal BERT-like model, MoBERT, which offers strongly human-correlated sample-level evaluations while maintaining near-perfect model-level correlation. Our results demonstrate that this new metric exhibits extensive benefits over all current alternatives.
BibTeX
@inproceedings{voas2023what,
  author = {Jordan Voas and Yili Wang and Qixing Huang and Raymond Mooney},
  title = {{What is the Best Automated Metric for Text to Motion Generation?}},
  booktitle = {SIGGRAPH Asia},
  year = {2023},
  doi = {10.1145/3610548.3618185},
}
Figure