TL;DR
This paper introduces "role-play prompting," a novel technique that significantly enhances the zero-shot reasoning capabilities of Large Language Models (LLMs). By having LLMs adopt specific personas, the method consistently outperforms standard zero-shot approaches across various reasoning benchmarks, suggesting role-play acts as an implicit trigger for more effective Chain-of-Thought reasoning.
What Problem Does This Paper Address?
Modern Large Language Models (LLMs) have demonstrated impressive abilities in understanding and generating text, but their reasoning capabilities, especially in zero-shot settings (without task-specific examples), remain an area for improvement. While techniques like Chain-of-Thought (CoT) prompting have shown success by encouraging step-by-step reasoning, the potential of role-playing – a known strength of LLMs – to boost reasoning performance has been underexplored. This paper investigates whether assigning LLMs specific roles can lead to better performance on complex reasoning tasks, aiming to find a more effective prompting strategy.
Methods
The authors propose a two-stage role-play prompting methodology.
- Prompt Construction: This stage involves designing two types of prompts:
- Role-Setting Prompt: A user-designed prompt that defines the specific role the LLM should adopt (e.g., a math teacher, a coin). This prompt is tailored to the task at hand, aiming to leverage the role's inherent advantages.
- Role-Feedback Prompt: The LLM's acknowledgment of the role-setting prompt. This prompt is generated by sampling multiple responses from the LLM and selecting the most representative and immersive one to further anchor the model in its persona.
- Question Answering: In the second stage, the role-setting prompt, the chosen role-feedback prompt, and the actual reasoning question are combined into a single prompt to query the LLM. This two-round dialogue process (first for role immersion, then for answering) is designed to maximize the LLM's engagement with its assigned role. The authors found that this two-stage approach, with complementary descriptions, enhances immersion and leads to better performance.
Datasets
The study evaluates the proposed method on 12 diverse reasoning benchmarks, categorized as:
- Arithmetic: MultiArith, GSM8K, AddSub, AQUA-RAT, SingleEq, SVAMP.
- Commonsense Reasoning: CSQA, StrategyQA.
- Symbolic Reasoning: Last Letter Concatenation, Coin Flip.
- Other: Date Understanding, Tracking Shuffled Objects (from BIG-bench).
No new datasets were introduced; the study utilizes existing, well-established benchmarks.
Key Insights and Ideas
- Role-Play as an Implicit CoT Trigger: The most significant insight is that role-play prompting acts as an effective implicit trigger for Chain-of-Thought (CoT) reasoning, even in cases where explicit "think step by step" instructions might not be as effective or are not spontaneously generated by the LLM.
- Enhanced Immersion Leads to Better Reasoning: The two-stage prompt structure, allowing the LLM to first elaborate on its role, significantly improves reasoning performance compared to simpler, single-turn prompts. This suggests that deeper role immersion is crucial.
- Advantaged Roles Yield Better Results: Selecting roles that naturally align with or have an advantage in the specific task (e.g., a math teacher for math problems) leads to superior performance.
- Generalization Across LLMs and Scales: The role-play prompting method demonstrates effectiveness not only on ChatGPT but also on open-source LLMs like Llama 2-Chat and Vicuna, and across different model sizes, indicating its broad applicability.
Key Results
- Superior Performance: Role-play prompting consistently outperforms the standard zero-shot baseline on 10 out of 12 datasets, achieving on-par performance on the remaining two.
- Significant Gains: For instance, on the AQuA dataset, accuracy increased from 53.5% to 63.8% with ChatGPT, and on Last Letter, it jumped from 23.8% to 84.2%.
- Outperforming Zero-Shot-CoT: The proposed method often surpasses Zero-Shot-CoT, especially in tasks where LLMs don't spontaneously generate CoT. While Zero-Shot-CoT shows improvement, role-play prompting often yields better results, particularly in reinforcing CoT.
- Competitive with Few-Shot: Despite being a zero-shot method, role-play prompting achieves performance comparable to, and sometimes exceeding, few-shot CoT methods.
Limitations and Future Work
- Manual Prompt Design: The current approach relies on manually designed role-setting and role-feedback prompts. This process is time-consuming and may not always guarantee optimal results.
- Future Work - Automated Role Selection: The authors suggest future research could focus on enabling LLMs to autonomously select appropriate roles and design prompts. This would make the method more scalable and extend its application to a wider range of domains beyond reasoning.
Summary updated