Introduction
The field of Text-to-Motion (T2M) generation—creating 3D human animations from natural language prompts like "a person walks in a circle and then sits"—is advancing rapidly. However, evaluating the quality and semantic alignment of these generated motions remains a significant challenge. Traditional metrics, such as Fréchet Inception Distance (FID) or L1 distance, compare a generated motion to a single "ground truth" reference. Because a single prompt can be correctly animated in an infinite number of ways, these metrics often unfairly penalize perfectly valid, creative motions.
Newer "reference-free" metrics attempt to solve this by training evaluators directly on 3D data. Unfortunately, they fall into a data-scarcity trap: high-quality 3D motion data is expensive and rare. Consequently, these evaluators often fail to generalize or can be easily "hacked" by models that produce high scores without achieving actual semantic alignment. To address this, researchers have introduced VeMo, a novel zero-shot framework that evaluates 3D human motions by rendering them into 2D videos and leveraging the vast, internet-scale knowledge of pre-trained Video-Language Models (VLMs).
The VeMo Framework
VeMo decouples motion evaluation from the scarcity of 3D data by transforming it into a 2D video reasoning task. The framework operates through four key steps:
- Rendering: Generated 3D motion sequences are rendered into 2D skinned human videos using tools like Blender.
- VLM as a Judge: Instead of training a bespoke 3D evaluator from scratch, VeMo utilizes pre-trained VLMs (such as InternVL). The VLM is provided with the rendered video and a prompt asking if the motion aligns with the text.
- Soft Scoring: Rather than relying on a binary "Yes/No" output, VeMo extracts the VLM's internal probability (likelihood) for affirmative tokens to generate a normalized, continuous alignment score.
- Entropy-Driven View Selection: Projecting 3D motion into 2D often introduces self-occlusion (e.g., an arm hidden behind the torso). VeMo renders the motion from multiple camera angles and calculates the predictive entropy for each. It automatically selects the view with the lowest entropy—essentially the perspective where the VLM is most "confident"—ensuring the evaluation relies on the most informative angle.
A New Meta-Evaluation Benchmark
To rigorously test how well different metrics perform, the researchers introduced a new test-only, human-annotated meta-evaluation benchmark.
- Source & Content: The benchmark utilizes prompts from the HumanML3D test set, comprising 2,202 text-motion pairs generated by representative T2M models (MDM and MotionGPT).
- Annotations: Human experts labeled each pair for Faithfulness (alignment with the text) and Naturalness (human-like movement).
- Purpose: Unlike previous datasets used to train evaluators, this benchmark provides a fair, standardized environment specifically for comparing evaluation metrics.
Key Insights and Results
The core innovation of VeMo is leveraging general intelligence: there is no need to teach a model what "walking" looks like in 3D if a VLM already understands it from millions of 2D videos. Furthermore, using Shannon entropy as a proxy for visibility brilliantly solves the camera-angle problem without requiring manual intervention.
Extensive experiments on the new benchmark yielded striking results:
- Superior Correlation: VeMo achieved a 4x improvement in correlation coefficients (Kendall’s $\tau$ and Spearman’s $\rho$) with human judgments compared to the best existing reference-free methods.
- Outperforming Reference-Based Metrics: Even without access to a ground-truth reference motion, VeMo's zero-shot approach correlated better with human evaluation than L1 distance. The study also confirmed that standard Multimodal (MM) Distance has almost zero correlation with human judgment, highlighting the unreliability of legacy metrics.
- Robustness and Efficiency: The framework remained effective even when prompts were paraphrased or when motions were rendered as simple stick figures. While high-quality "skinned" rendering performs best, it introduces a bottleneck (~30s per motion). However, VLM inference is fast (~1.6s), and a stick-figure rendering pipeline can process motions in under a second for rapid screening.
Limitations and Future Directions
While VeMo represents a significant leap forward, it has limitations. High-quality rendering is computationally more expensive than simple mathematical distance checks between 3D joints. Additionally, VeMo inherits any semantic biases or perceptual limits of the underlying VLM.
Currently, VeMo provides a coarse alignment score. Future work could focus on fine-grained diagnostics—identifying exactly which body part or which specific second of the animation failed to match the text. Finally, the current benchmark is limited to the joint set of HumanML3D; evaluating dexterous hand movements or subtle facial expressions will require more complex rendering techniques and even more advanced VLMs.
Summary updated