Zero-Shot Text-to-Motion Evaluation using Video Language Models

Yuwen Ji, Donglin Wang, Yue ZhangICML 2026 · OpenReview

Introduction

The field of Text-to-Motion (T2M) generation—creating 3D human animations from natural language prompts like "a person walks in a circle and then sits"—is advancing rapidly. However, evaluating the quality and semantic alignment of these generated motions remains a significant challenge. Traditional metrics, such as Fréchet Inception Distance (FID) or L1 distance, compare a generated motion to a single "ground truth" reference. Because a single prompt can be correctly animated in an infinite number of ways, these metrics often unfairly penalize perfectly valid, creative motions.

Newer "reference-free" metrics attempt to solve this by training evaluators directly on 3D data. Unfortunately, they fall into a data-scarcity trap: high-quality 3D motion data is expensive and rare. Consequently, these evaluators often fail to generalize or can be easily "hacked" by models that produce high scores without achieving actual semantic alignment. To address this, researchers have introduced VeMo, a novel zero-shot framework that evaluates 3D human motions by rendering them into 2D videos and leveraging the vast, internet-scale knowledge of pre-trained Video-Language Models (VLMs).

The VeMo Framework

VeMo decouples motion evaluation from the scarcity of 3D data by transforming it into a 2D video reasoning task. The framework operates through four key steps:

  1. Rendering: Generated 3D motion sequences are rendered into 2D skinned human videos using tools like Blender.
  2. VLM as a Judge: Instead of training a bespoke 3D evaluator from scratch, VeMo utilizes pre-trained VLMs (such as InternVL). The VLM is provided with the rendered video and a prompt asking if the motion aligns with the text.
  3. Soft Scoring: Rather than relying on a binary "Yes/No" output, VeMo extracts the VLM's internal probability (likelihood) for affirmative tokens to generate a normalized, continuous alignment score.
  4. Entropy-Driven View Selection: Projecting 3D motion into 2D often introduces self-occlusion (e.g., an arm hidden behind the torso). VeMo renders the motion from multiple camera angles and calculates the predictive entropy for each. It automatically selects the view with the lowest entropy—essentially the perspective where the VLM is most "confident"—ensuring the evaluation relies on the most informative angle.

A New Meta-Evaluation Benchmark

To rigorously test how well different metrics perform, the researchers introduced a new test-only, human-annotated meta-evaluation benchmark.

  • Source & Content: The benchmark utilizes prompts from the HumanML3D test set, comprising 2,202 text-motion pairs generated by representative T2M models (MDM and MotionGPT).
  • Annotations: Human experts labeled each pair for Faithfulness (alignment with the text) and Naturalness (human-like movement).
  • Purpose: Unlike previous datasets used to train evaluators, this benchmark provides a fair, standardized environment specifically for comparing evaluation metrics.

Key Insights and Results

The core innovation of VeMo is leveraging general intelligence: there is no need to teach a model what "walking" looks like in 3D if a VLM already understands it from millions of 2D videos. Furthermore, using Shannon entropy as a proxy for visibility brilliantly solves the camera-angle problem without requiring manual intervention.

Extensive experiments on the new benchmark yielded striking results:

  • Superior Correlation: VeMo achieved a 4x improvement in correlation coefficients (Kendall’s $\tau$ and Spearman’s $\rho$) with human judgments compared to the best existing reference-free methods.
  • Outperforming Reference-Based Metrics: Even without access to a ground-truth reference motion, VeMo's zero-shot approach correlated better with human evaluation than L1 distance. The study also confirmed that standard Multimodal (MM) Distance has almost zero correlation with human judgment, highlighting the unreliability of legacy metrics.
  • Robustness and Efficiency: The framework remained effective even when prompts were paraphrased or when motions were rendered as simple stick figures. While high-quality "skinned" rendering performs best, it introduces a bottleneck (~30s per motion). However, VLM inference is fast (~1.6s), and a stick-figure rendering pipeline can process motions in under a second for rapid screening.

Limitations and Future Directions

While VeMo represents a significant leap forward, it has limitations. High-quality rendering is computationally more expensive than simple mathematical distance checks between 3D joints. Additionally, VeMo inherits any semantic biases or perceptual limits of the underlying VLM.

Currently, VeMo provides a coarse alignment score. Future work could focus on fine-grained diagnostics—identifying exactly which body part or which specific second of the animation failed to match the text. Finally, the current benchmark is limited to the joint set of HumanML3D; evaluating dexterous hand movements or subtle facial expressions will require more complex rendering techniques and even more advanced VLMs.

Summary updated

Abstract

Text-to-motion (T2M) generation has become a fundamental task, yet existing evaluation metrics often fail to capture whether a generated motion semantically matches its text description. We propose VeMo, a zero-shot evaluation framework that renders generated human motions into videos and uses pretrained video-language models (VLMs) to assess text-motion alignment. Instead of training an evaluator on scarce motion-specific labels, VeMo transfers the semantic reasoning ability of VLMs to T2M evaluation through normalized likelihood-based scoring. To reduce the effect of 3D-to-2D projection ambiguity, we introduce an entropy-driven uncertainty analysis for identifying reliable rendered views. To address the lack of rigorous standards in the field, we further contribute a test-only and human-annotated meta-evaluation benchmark, covering motions generated by multiple representative T2M models. Extensive experiments show that VeMo correlates better with human judgments than existing reference-based and reference-free metrics. Additional analyses on view selection, rendering protocols, textual prompt robustness, and computational trade-offs characterize both the promise and limitations of VLM-based T2M evaluation.
BibTeX
@inproceedings{ji2026zero,
  author = {Yuwen Ji and Donglin Wang and Yue Zhang},
  title = {{Zero-Shot Text-to-Motion Evaluation using Video Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=Sf8ubkiEkW},
}
Figure