Playpen: An Environment for Exploring Learning From Dialogue Game Feedback

Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia2025 · DOI

Abstract: Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

2025.

Introduction

Post-training Large Language Models (LLMs) typically relies on Reinforcement Learning from Human Feedback (RLHF), which is often subjective and costly, or verifiable rewards (e.g., code execution), which are objective but limited to formal domains. A significant gap exists in training models for interactive communication—multi-turn settings where success depends on the outcome of a conversation. Most current methods evaluate whether a response looks correct rather than if it effectively achieves a goal.

To address this, researchers introduced PLAYPEN, an environment utilizing Dialogue Game Feedback (DGF). DGF provides a signal that is objective (programmatically calculated), teleological (goal-directed), and inherently multi-turn, offering a robust middle ground for training communicative agents.

The PLAYPEN Environment and Methodology

PLAYPEN, built on the clembench framework, allows for the exploration of various training regimes using dialogue games like Taboo, Wordle, and Codenames. The study categorizes feedback and learning methods to understand how different signals affect model performance and general intelligence.

Taxonomy of feedback sources and learning methods, highlighting Dialogue Game Feedback as a distinct objective source.

The researchers evaluated three primary training approaches:

  • Supervised Fine-Tuning (SFT): An imitation learning approach using a "Cold Start" dataset of ~7,000 successful game transcripts.
  • Direct Preference Optimization (DPO): An offline alignment method where models learn to prefer successful trajectories over unsuccessful ones.
  • Group Relative Policy Optimization (GRPO): An online reinforcement learning approach where models play in real-time against a teacher (GPT-4o-mini) and receive programmatic rewards based on game rules and outcomes.

Experimental Results

The study utilized 15 dialogue games, with seven used for training (in-domain) and eight held out for testing (out-of-domain). Performance was measured using the "Clemscore," which aggregates rule-following and task success.

Performance of Llama-3.1 and Qwen-2 models across different training regimes for in-domain and out-of-domain games.

Key Findings

  • The "Alignment Tax": While SFT significantly improved in-domain performance (e.g., Llama-3.1-8B jumping from 19.39 to 40.11), it often led to a regression in general instruction-following. For instance, IFEval scores dropped by nearly 10 points, suggesting the model learned game formats at the expense of general utility.
  • Generalization via Online Learning: GRPO emerged as the superior method for generalization. Unlike SFT, which saw performance drops on unseen games, GRPO improved out-of-domain scores (from 24.58 to 33.92 for the 8B model) while maintaining high general benchmark performance.
  • Strategic Reasoning: Dialogue games require managing uncertainty and coordination, distinct from the formal logic required for math or coding tasks. DGF effectively captures this strategic dimension.

Conclusion and Future Work

PLAYPEN demonstrates that objective, goal-directed feedback from dialogue games can enhance an LLM's interactive capabilities. While imitation learning provides quick gains, online reinforcement learning via GRPO better preserves general intelligence and fosters strategic generalization. Future research will focus on scaling these methods to larger models and developing more granular "intermediate" feedback to improve credit assignment in complex multi-turn interactions.

Summary updated

BibTeX
@misc{horst2025playpen,
  author = {Nicola Horst and Davide Mazzaccara and Antonia Schmidt and Michael Sullivan and Filippo Momentè and Luca Franceschetti and Philipp Sadler and Sherzod Hakimov and Alberto Testoni and Raffaella Bernardi and Raquel Fernández and Alexander Koller and Oliver Lemon and David Schlangen and Mario Giulianelli and Alessandro Suglia},
  title = {{Playpen: An Environment for Exploring Learning From Dialogue Game Feedback}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1517},
}
Figure