RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models

Noah Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, Junran Peng

Introduction

While state-of-the-art Large Language Models (LLMs) like GPT-4 exhibit impressive role-playing capabilities, they remain closed-source, costly, and restricted by context window limitations. Open-source alternatives often struggle with fine-grained character imitation, typically defaulting to coarse-grained personas (e.g., "helpful assistant") rather than specific characters like Sherlock Holmes or Twilight Sparkle. RoleLLM addresses this gap by introducing a comprehensive framework to evaluate and enhance the character-level role-playing abilities of open-source LLMs.

The RoleLLM Framework

The authors propose a four-stage pipeline designed to elicit nuanced role-playing behavior and inject character-specific episodic memory into open-source models.

Overview of the RoleLLM framework, including data construction (Context-Instruct, RoleGPT) and instruction tuning (RoCIT).
  1. Role Profile Construction: Detailed profiles for 100 characters (95 English, 5 Chinese) were built using scripts from movies and TV shows, capturing descriptions, catchphrases, and structured dialogues.
  2. Context-Instruct: To provide models with character-specific knowledge, LLMs generated question-answer pairs from script segments. A confidence score was used to filter out hallucinations, ensuring high-quality episodic memory injection.
  3. RoleGPT: GPT-4 was utilized to generate training data focused on speaking styles. By employing few-shot dialogue engineering—formatting prompts as conversational histories—the authors achieved superior stylistic fidelity compared to standard prompt engineering.
  4. Role-Conditioned Instruction Tuning (RoCIT): Open-source models (LLaMA and ChatGLM) were fine-tuned on this data. Instead of relying on heavy retrieval-augmented generation (RAG) during inference, the models were trained to adopt personas via concise system instructions.

RoleBench: A Fine-Grained Benchmark

A central contribution of this work is RoleBench, the first systematic benchmark for fine-grained character-level role-playing. It comprises over 168,000 samples across 100 roles, categorized into general instructions (stylistic imitation) and role-specific instructions (episodic memory).

Statistical breakdown of the RoleBench dataset, highlighting the scale of roles and instructions.

Key Insights and Results

The experimental results yielded several critical findings for the development of role-playing agents:

  • Dialogue Engineering vs. Prompt Engineering: Conversational history formatting is significantly more effective at eliciting stylistic nuances from GPT-4 than standard prompting.
  • The Limits of Retrieval Augmentation: For smaller models like LLaMA-7B, RAG often introduced noise that degraded performance. System-instruction-based fine-tuning proved more robust and context-efficient.
  • Performance Parity: The fine-tuned RoleLLaMA (English) and RoleGLM (Chinese) models significantly outperformed general instruction-tuned models like Alpaca and Vicuna. Notably, RoleLLaMA achieved results comparable to the GPT-4-based RoleGPT and even surpassed it in role-specific knowledge.
  • Generalization: RoleLLaMA demonstrated strong zero-shot generalization to unseen roles, successfully adopting new personas based solely on brief system descriptions.

Conclusion and Future Work

RoleLLM demonstrates that open-source models can achieve high-fidelity role-playing through targeted instruction tuning and synthetic data generation. While currently focused on single-turn interactions and limited to English and Chinese, future work aims to expand to multi-turn conversations and a broader range of languages. Additionally, the authors highlight the importance of safety moderation, as role-playing can potentially be used to bypass standard LLM safety filters.

Summary updated

Abstract

The advent of Large Language Models (LLMs) has paved the way for complex tasks such as role-playing, which enhances user interactions by enabling models to imitate various characters. However, the closed-source nature of state-of-the-art LLMs and their general-purpose training limit role-playing optimization. In this paper, we introduce RoleLLM, a framework to benchmark, elicit, and enhance role-playing abilities in LLMs. RoleLLM comprises four stages: (1) Role Profile Construction for 100 roles; (2) Context-Based Instruction Generation (Context-Instruct) for role-specific knowledge extraction; (3) Role Prompting using GPT (RoleGPT) for speaking style imitation; and (4) Role-Conditioned Instruction Tuning (RoCIT) for fine-tuning open-source models along with role customization. By Context-Instruct and RoleGPT, we create RoleBench, the first systematic and fine-grained character-level benchmark dataset for role-playing with 168,093 samples. Moreover, RoCIT on RoleBench yields RoleLLaMA (English) and RoleGLM (Chinese), significantly enhancing role-playing abilities and even achieving comparable results with RoleGPT (using GPT-4).
BibTeX
@misc{wang2023rolellm,
  author = {Noah Wang and Zhongyuan Peng and Haoran Que and Jiaheng Liu and Wangchunshu Zhou and Yuhan Wu and Hongcheng Guo and Ruitong Gan and Zehao Ni and Jian Yang and Man Zhang and Zhaoxiang Zhang and Wanli Ouyang and Ke Xu and Wenhao Huang and Jie Fu and Junran Peng},
  title = {{RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models}},
  year = {2023},
  eprint = {2310.00746},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00746},
}
Figure