Better Zero-Shot Reasoning with Role-Play Prompting

Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, Xiaohang Dong

TL;DR

This paper introduces "role-play prompting," a novel technique that significantly enhances the zero-shot reasoning capabilities of Large Language Models (LLMs). By having LLMs adopt specific personas, the method consistently outperforms standard zero-shot approaches across various reasoning benchmarks, suggesting role-play acts as an implicit trigger for more effective Chain-of-Thought reasoning.

What Problem Does This Paper Address?

Modern Large Language Models (LLMs) have demonstrated impressive abilities in understanding and generating text, but their reasoning capabilities, especially in zero-shot settings (without task-specific examples), remain an area for improvement. While techniques like Chain-of-Thought (CoT) prompting have shown success by encouraging step-by-step reasoning, the potential of role-playing – a known strength of LLMs – to boost reasoning performance has been underexplored. This paper investigates whether assigning LLMs specific roles can lead to better performance on complex reasoning tasks, aiming to find a more effective prompting strategy.

Methods

The authors propose a two-stage role-play prompting methodology.

  1. Prompt Construction: This stage involves designing two types of prompts:
  2. Role-Setting Prompt: A user-designed prompt that defines the specific role the LLM should adopt (e.g., a math teacher, a coin). This prompt is tailored to the task at hand, aiming to leverage the role's inherent advantages.
  3. Role-Feedback Prompt: The LLM's acknowledgment of the role-setting prompt. This prompt is generated by sampling multiple responses from the LLM and selecting the most representative and immersive one to further anchor the model in its persona.
  4. Question Answering: In the second stage, the role-setting prompt, the chosen role-feedback prompt, and the actual reasoning question are combined into a single prompt to query the LLM. This two-round dialogue process (first for role immersion, then for answering) is designed to maximize the LLM's engagement with its assigned role. The authors found that this two-stage approach, with complementary descriptions, enhances immersion and leads to better performance.

Datasets

The study evaluates the proposed method on 12 diverse reasoning benchmarks, categorized as:

  • Arithmetic: MultiArith, GSM8K, AddSub, AQUA-RAT, SingleEq, SVAMP.
  • Commonsense Reasoning: CSQA, StrategyQA.
  • Symbolic Reasoning: Last Letter Concatenation, Coin Flip.
  • Other: Date Understanding, Tracking Shuffled Objects (from BIG-bench).

No new datasets were introduced; the study utilizes existing, well-established benchmarks.

Key Insights and Ideas

  • Role-Play as an Implicit CoT Trigger: The most significant insight is that role-play prompting acts as an effective implicit trigger for Chain-of-Thought (CoT) reasoning, even in cases where explicit "think step by step" instructions might not be as effective or are not spontaneously generated by the LLM.
  • Enhanced Immersion Leads to Better Reasoning: The two-stage prompt structure, allowing the LLM to first elaborate on its role, significantly improves reasoning performance compared to simpler, single-turn prompts. This suggests that deeper role immersion is crucial.
  • Advantaged Roles Yield Better Results: Selecting roles that naturally align with or have an advantage in the specific task (e.g., a math teacher for math problems) leads to superior performance.
  • Generalization Across LLMs and Scales: The role-play prompting method demonstrates effectiveness not only on ChatGPT but also on open-source LLMs like Llama 2-Chat and Vicuna, and across different model sizes, indicating its broad applicability.

Key Results

  • Superior Performance: Role-play prompting consistently outperforms the standard zero-shot baseline on 10 out of 12 datasets, achieving on-par performance on the remaining two.
  • Significant Gains: For instance, on the AQuA dataset, accuracy increased from 53.5% to 63.8% with ChatGPT, and on Last Letter, it jumped from 23.8% to 84.2%.
  • Outperforming Zero-Shot-CoT: The proposed method often surpasses Zero-Shot-CoT, especially in tasks where LLMs don't spontaneously generate CoT. While Zero-Shot-CoT shows improvement, role-play prompting often yields better results, particularly in reinforcing CoT.
  • Competitive with Few-Shot: Despite being a zero-shot method, role-play prompting achieves performance comparable to, and sometimes exceeding, few-shot CoT methods.

Limitations and Future Work

  • Manual Prompt Design: The current approach relies on manually designed role-setting and role-feedback prompts. This process is time-consuming and may not always guarantee optimal results.
  • Future Work - Automated Role Selection: The authors suggest future research could focus on enabling LLMs to autonomously select appropriate roles and design prompts. This would make the method more scalable and extend its application to a wider range of domains beyond reasoning.

Summary updated

Abstract

Modern large language models (LLMs) exhibit a remarkable capacity for role-playing, enabling them to embody not only human characters but also non-human entities. This versatility allows them to simulate complex human-like interactions and behaviors within various contexts, as well as to emulate specific objects or systems. While these capabilities have enhanced user engagement and introduced novel modes of interaction, the influence of role-playing on LLMs' reasoning abilities remains underexplored. In this study, we introduce a strategically designed role-play prompting methodology and assess its performance under the zero-shot setting across twelve diverse reasoning benchmarks. Our empirical results illustrate that role-play prompting consistently surpasses the standard zero-shot approach across most datasets. Notably, in experiments conducted using ChatGPT, accuracy on AQuA rises from 53.5% to 63.8%, and on Last Letter from 23.8% to 84.2%.Upon further comparison with the Zero-Shot-CoT technique, which prompts the model to "think step by step", our study demonstrates that role-play prompting acts as a more effective trigger for the CoT process. This highlights its potential to augment the reasoning capabilities of LLMs. We release our code at https://github.com/NKU-HLT/Role-Play-Prompting.
BibTeX
@inproceedings{kong2024better,
  author = {Aobo Kong and Shiwan Zhao and Hao Chen and Qicheng Li and Yong Qin and Ruiqi Sun and Xin Zhou and Enzhi Wang and Xiaohang Dong},
  title = {{Better Zero-Shot Reasoning with Role-Play Prompting}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.228},
}
Figure