Adapting Vision-Language Models for Evaluating World Models

Mariya Hendriksen, Tabish Rashid, David Bignell, Raluca Stevenson, Abdelhak Lemkhenter, Katja Hofmann, Sam Devlin, Sarah Parisot

Abstract

World models -- generative models that simulate environment dynamics conditioned on past observations and actions -- are gaining prominence in planning, simulation, and embodied AI. However, evaluating their rollouts remains a fundamental challenge, requiring fine-grained, temporally grounded assessment of action alignment and semantic consistency -- capabilities not captured by existing metrics. Vision-Language Models (VLMs) have shown promise as automatic evaluators of generative content due to their strong multimodal reasoning abilities. Yet, their use in fine-grained, temporally sensitive evaluation tasks remains limited and requires targeted adaptation. We introduce an evaluation protocol targeting two recognition tasks -- action recognition and character recognition -- each assessed across binary, multiple-choice, and open-ended formats. To support this, we present UNIVERSE (UNIfied Vision-language Evaluator for Rollouts in Simulated Environments), a method for adapting VLMs to rollout evaluation under data and compute constraints. In our extensive experiments totaling over 5,154 GPU-days, we explore full, partial, and parameter-efficient adaptation methods across various task formats, context lengths, sampling methods, and data compositions. The resulting evaluator achieves parity with task-specific baselines using a single unified checkpoint. Human studies across seven diverse environments confirm strong alignment with human judgments, establishing UNIVERSE as a lightweight, adaptable, and semantics-aware evaluator for world models.
BibTeX
@inproceedings{hendriksen2025adapting,
  author = {Mariya Hendriksen and Tabish Rashid and David Bignell and Raluca Stevenson and Abdelhak Lemkhenter and Katja Hofmann and Sam Devlin and Sarah Parisot},
  title = {{Adapting Vision-Language Models for Evaluating World Models}},
  booktitle = {LAW @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=h4lBkLlYHg},
}
Figure