TL;DR
This paper introduces a scalable, collaborative multi-agent framework designed to automatically generate high-quality Murder Mystery game scripts to train Vision-Language Models (VLMs). By combining Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO) reinforcement learning monitored by an automated "ScoreAgent," the authors significantly enhance a VLM's ability to handle multi-hop reasoning, uncertainty, and strategic deception under imperfect-information conditions.
What Problem Does This Paper Address?
While Vision-Language Models (VLMs) excel at standard perceptual tasks like image captioning and visual question answering, they frequently stumble when dropped into complex, multi-player environments that require multi-hop reasoning under imperfect or deceptive information. Real-world scenarios—such as judicial proceedings or business negotiations—require agents to interact dynamically, evaluate credibility, and piece together fragmented, sometimes intentionally misleading, textual and visual clues.
To study this, the authors use Murder Mystery games as a proxy environment. However, creating large-scale, high-quality datasets containing intricate plots, character backstories, and multimodal clues is incredibly expensive and time-consuming for humans. This paper addresses the lack of robust training data and evaluation frameworks for multi-modal, multi-hop reasoning under uncertainty.
Methods
The authors propose a two-pronged solution: an agent-driven data generation pipeline and a two-stage agent-monitored training strategy.
1. Multi-Agent Script Synthesis Framework
The framework divides responsibilities between Generation Agents and Evaluation Agents to collaboratively produce coherent game scripts:
- OutlineAgent & CharacterAgent: Create the overarching crime narrative, timelines, character profiles, motives, and hidden secrets.
- ClueAgent: Generates a mix of textual and visual clues (e.g., maps, text-dense notes) that aid deduction without prematurely revealing the killer.
- RoleplayAgent & QaAgent: Simulate multi-turn dialogues (self-introductions and discussions) and construct multi-hop question-answer pairs.
- CriticAgent: Acts as a quality control mechanism, evaluating the generated plots for complexity, difficulty, and logical consistency, providing feedback loops for refinement.
2. Two-Stage Agent-Monitored Training
Once the data is synthesized, the target VLM undergoes a specialized training pipeline:
- Stage 1: Supervised Fine-Tuning (SFT): The model is fine-tuned using LoRA on the synthetic dataset to establish baseline role-playing capabilities and structured Chain-of-Thought (CoT) reasoning patterns.
- Stage 2: Reinforcement Learning via GRPO: To refine strategic behaviors (like deception for the murderer or cooperation for the innocent), the authors implement Group Relative Policy Optimization (GRPO). Instead of a static reward model, they employ a ScoreAgent (an LLM-as-Judge). The ScoreAgent dynamically scores model outputs based on role consistency, formatting, and clue matching, penalizing rule-breaking behaviors like premature identity revelation or off-topic responses.
Datasets
The pipeline automatically generated a new synthetic dataset consisting of:
- 34 unique Murder Mystery scripts serving as the narrative backbones.
- 1,060 long-script-based QA pairs.
- 2,725 multimodal QA pairs (split into 1,249 text-rich and 1,476 media-rich instances).
For evaluation, the authors utilized WhodunitBench, an existing human-annotated benchmark containing 50 murder mystery scripts, ensuring no circular evaluation risks since the evaluation set remained entirely human-curated.
Key Insights and Ideas
- Deception as a Trainable Feature: Unlike standard reasoning tasks where the goal is absolute truthfulness, Murder Mystery games require modeling strategic deception (for the murderer) and lie detection (for the innocents).
- LLM-as-Judge for Unverifiable Tasks: Dialogue and role-play do not have single "correct" ground-truth answers. Using an LLM-as-Judge (ScoreAgent) to provide relative reward signals within a rollout group allows the model to optimize for subjective criteria like "naturalness" and "strategic value."
- Data Complementarity: Ablation studies revealed that training on a mix of human-annotated and synthetic data yields the best performance. Human data provides high-quality grounding, while synthetic data enriches interaction diversity and complex reasoning paths.
Key Results
The framework was evaluated using Qwen2.5-VL (3B and 7B variants) as backbones. The enhanced models demonstrated massive performance leaps across seven core metrics, including Multi-hop Multimodal Reasoning (MMR) and Decision-Making (DM):
- Qwen2.5-VL-7B-Instruct (Full Model): Achieved an MMR score of 58.42, outperforming its baseline (37.63) by +20.79 points and surpassing proprietary models like Gemini-1.5-Pro (57.39) and Claude (57.78).
- Decision-Making (DM): The 7B model's accuracy in correctly voting for the murderer rose to 36.87%, a significant improvement over the baseline's 25.61%.
- Reward Convergence: Reinforcement learning curves showed that verifiable tasks (QA) converged to stable, high rewards, while role-playing tasks exhibited expectedly higher variance due to the subjective nature of dialogue.
Limitations and Future Work
Despite its success, the authors note a few limitations:
- Partial Human Reliance: The pipeline is not 100% automated; it still requires partial human verification during the image-clue alignment phase of WhodunitBench.
- Simplified Environment: A simulated murder mystery game simplifies the highly unstructured, dynamic nature of real-world scenarios like legal trials or corporate negotiations.
Future work aims to expand this methodology into more complex, unstructured real-world domains and establish clearer ethical guidelines for training models in strategic deception.
Summary updated