DVM: Towards Controllable LLM Agents in Social Deduction Games

Hao WangICASSP 2025 · OpenReview

TL;DR

This paper introduces DVM, a novel framework for creating LLM agents that can dynamically adjust their skill level in social deduction games (SDGs) like Werewolf. By integrating reinforcement learning with a unique reward mechanism that constrains the agent's win rate, DVM agents can be controlled to perform at specific proficiency levels, offering more adaptive and balanced gameplay.

What Problem Does This Paper Address?

Social deduction games (SDGs) like Werewolf are complex environments where success hinges on conversation, inference, and strategic decision-making. While Large Language Models (LLMs) have shown impressive capabilities in these games, existing agents primarily focus on maximizing their performance (i.e., achieving the highest possible win rate). This leaves a gap in controlling the proficiency of these agents. The ability to adjust an agent's skill level is crucial for practical applications like dynamic difficulty scaling in games, ensuring fairness, and enabling safety research. This paper addresses the need for LLM agents that can be controllably tuned to achieve specific win rates, rather than just optimal performance.

Methods

DVM is a three-component framework:

  • Predictor: This LLM-based module analyzes player interactions and voting patterns to infer the roles and relationships of other players. It outputs these predictions in natural language, which are then parsed for the Decider.
  • Decider: This module makes strategic decisions (e.g., voting, saving, killing) based on the current game state, the Predictor's insights, and a win rate constraint. It uses the PPO (Proximal Policy Optimization) algorithm for training, a standard for stable reinforcement learning in complex environments.
  • Discussor: This LLM module generates contextually appropriate and strategic dialogue to influence other players, based on the Decider's actions and the game's context.

The core innovation lies in the training methodology:

  • Supervised Training: Initially, the Predictor and Decider are fine-tuned on a dataset of human-player games (FanLang-9) to understand game mechanics and player strategies.
  • Reinforcement Learning:
    • The Predictor is further optimized using Direct Preference Optimization (DPO) with its predictions and correct answers as samples.
  • The Decider is trained using PPO.
  • Decision Chain Reward: Instead of rewarding individual actions, DVM uses a reward that evaluates entire sequences of decisions. This is achieved by analyzing decision chains from the training data and associating them with win rates.
  • Win Rate-Constrained Reward: To control proficiency, a novel reward function is introduced. It calculates the deviation between the agent's actual win rate and a specified target win rate. This deviation is then used to generate a controlled reward (cr_ctrl.) that encourages the agent to stabilize its performance around the target win rate. This reward is combined with a step reward to form the total reward.

Datasets

  • FanLang-9 Dataset: This dataset, consisting of over 18,000 human-player game records, is used for both supervised training of the Predictor and Decider, and for analyzing decision chains to derive win rate associations.

Key Insights and Ideas

  • Controllability is Key: The paper argues that simply maximizing performance isn't enough; controllable agents are essential for practical game development and AI safety research.
  • Decision Chains Matter: Evaluating entire sequences of decisions, rather than isolated actions, provides a more holistic view of an agent's strategic sophistication and its impact on game outcomes.
  • Win Rate as a Control Knob: By directly incorporating a win rate constraint into the reward function, agents can be "tuned" to perform at specific skill levels, offering a novel way to manage AI difficulty.
  • LLMs as Multi-faceted Agents: DVM demonstrates that LLMs can effectively handle prediction, decision-making, and communication within a complex interactive environment.

Key Results

  • Superior Prediction: DVM's Predictor module outperforms existing methods (Random, GPT-3.5, GPT-4, LtM, Thinker) in both Werewolf prediction (identifying werewolves) and general identity prediction tasks, achieving higher ACC@N scores.
  • Effective Controllability: Experiments show that DVM agents can adjust their performance to meet specified win rate targets. While there's a gap between target and achieved win rates, DVM exhibits a clear upward trend in actual win rates as the constraints increase, unlike other methods that remain insensitive to win rate constraints.
  • Overall Performance: In unrestricted Werewolf games, DVM outperforms other methods like ReAct, GPT-3.5, GPT-4, LtM, and Thinker in terms of win rates across different roles (Werewolf, Villager, Other Roles).
  • Ablation Study: The ablation study confirms the effectiveness of DVM's core components, showing that both the Decision Chain Reward (DCR) and the Predictor contribute significantly to the overall performance.

Limitations and Future Work

  • Gap in Win Rate Achievement: While DVM shows controllability, there remains a gap between the specified win rate constraints and the actual achieved win rates. The paper notes that achieving higher target win rates becomes increasingly challenging.
  • Future Directions: The authors suggest that this work opens avenues for LLM agents' adaptive and balanced gameplay in SDGs and provides new insights into the development of controllable LLM agents. Future work could focus on closing the gap in win rate achievement and exploring DVM's applicability to other complex social deduction games.

Summary updated

Abstract

Large Language Models (LLMs) have advanced the capability of game agents in social deduction games (SDGs). These games rely heavily on conversation-driven interactions and require agents to infer, make decisions, and express based on such information. While this progress leads to more sophisticated and strategic non-player characters (NPCs) in SDGs, there exists a need to control the proficiency of these agents. This control not only ensures that NPCs can adapt to varying difficulty levels during gameplay, but also provides insights into the safety and fairness of LLM agents. In this paper, we present DVM, a novel framework for developing controllable LLM agents for SDGs, and demonstrate its implementation on one of the most popular SDGs, Werewolf. DVM comprises three main components: Predictor, Decider, and Discussor. By integrating reinforcement learning with a win rate-constrained decision chain reward mechanism, we enable agents to dynamically adjust their gameplay proficiency to achieve specified win rates. Experiments show that DVM not only outperforms existing methods in the Werewolf game, but also successfully modulates its performance levels to meet predefined win rate targets. These results pave the way for LLM agents’ adaptive and balanced gameplay in SDGs, opening new avenues for research in controllable game agents.
BibTeX
@inproceedings{wang2025dvm,
  author = {Hao Wang},
  title = {{DVM: Towards Controllable LLM Agents in Social Deduction Games}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=fXL4sy2Hcr},
}
Figure