clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents

Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, David SchlangenEMNLP 2023 · DOI

Abstract

Traditional Large Language Model (LLM) benchmarks are often static, failing to assess the "situated" intelligence required for effective conversational agency. We present clembench, a novel evaluation framework that utilizes interactive, language-based games to probe the ability of chat-optimized LLMs (cLLMs) to follow complex rules, track conversational state, and adapt to interlocutors. Our findings indicate that while advanced models like GPT-4 exhibit impressive strategic play, substantial challenges remain in rule adherence and feedback integration compared to human performance.

Introduction

Most LLM evaluations focus on question-answering or static text generation. However, as LLMs are increasingly deployed as conversational agents, there is a critical need to evaluate their functional language use in dynamic contexts. A model might possess vast factual knowledge yet fail to adhere to simple constraints in a game of Taboo. clembench (the Chat-optimized Language Model Benchmark) addresses this by treating game play as a proxy for conversational agency, requiring models to operate within a "situated" environment where every turn impacts the future state of the interaction.

The clemgame Framework

The researchers developed the clemgame framework, a Python-based environment where a programmatic "Game Master" (GM) mediates between two players. These players can be two LLMs (self-play) or an LLM and a deterministic bot.

The clemgame framework interaction loop.

The framework utilizes prompt-based scaffolding to assign "personas" and rules. The GM manages the turn-based interaction, ensuring that only valid, parsed responses are exchanged. Performance is measured via the clemscore, a macro-metric combining the percentage of successfully completed games (% Played) with a Quality Score reflecting the model's strategic success.

Benchmark Games

The initial benchmark, clembench v1.0, includes 250 instances across five dialogue games designed to test different cognitive and linguistic capabilities:

  1. Taboo: Describing a target word while avoiding specific "taboo" terms.
  2. Wordle: Guessing a 5-letter word using semantic clues and letter-based feedback.
  3. Drawing: Reconstructing a grid-based image from a text description.
  4. Picture Reference: Identifying a specific grid among distractors.
  5. Scorekeeping: Tracking shared versus private information in a travel-planning scenario.
An example of a Taboo game instance in clembench.

These games test various facets of agency, from spatial reasoning to the ability to handle negative constraints (e.g., "Don't say the word journey").

Key Results and Insights

Evaluation of models including GPT-4, Claude, and various open-source alternatives revealed several key insights:

  • Instruction Tuning is Critical: The transition from base models to chat-optimized versions (via RLHF) is essential for the agency required to follow game rules. Most open-source models struggled to even complete the games due to formatting failures.
  • The Challenge of Negative Constraints: Models frequently fail at Taboo by inadvertently using forbidden words, highlighting a persistent weakness in strictly adhering to negative instructions.
  • Feedback Integration: GPT-4 stands alone in its ability to consistently integrate multi-turn feedback (e.g., "yellow/green" letters in Wordle) to refine subsequent actions.
  • Performance Hierarchy: GPT-4 currently leads the benchmark with a clemscore of approximately 60. There is a clear correlation between a model's sophistication and its game-playing ability, yet even top models have not saturated the benchmark.

Conclusion

While clembench provides a rigorous test of conversational agency, it currently faces limitations such as being English-only and having a relatively small sample size per game. Future developments aim to support more players, integrate multimodal inputs, and enable human-AI competitive play. The fact that even GPT-4 only reaches a quality score of ~60% suggests that clembench will remain a challenging and relevant benchmark as LLM capabilities continue to evolve toward true conversational agency.

Summary updated

BibTeX
@inproceedings{chalamalasetti2023clembench,
  author = {Kranti Chalamalasetti and Jana Götze and Sherzod Hakimov and Brielen Madureira and Philipp Sadler and David Schlangen},
  title = {{clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.689},
}
Figure