TL;DR
This paper introduces "Storytelling Social Chatbots" (SSCs) that leverage Large Language Models (LLMs) to create engaging and believable fictional characters within online communities. By integrating character narratives and community interaction, SSCs aim to foster deeper social connections and enhance user engagement beyond traditional chatbots.
What Problem Does This Paper Address?
The paper addresses the challenge of creating social chatbots (SCs) that can foster meaningful interactions within community settings, moving beyond simple dyadic conversations. While LLMs have advanced SC capabilities, existing research has largely overlooked the potential of fictional narratives and characters to enhance social engagement and believability in a multi-user context. The authors question whether fictional characters can truly become integrated members of human society and how their stories influence perception.
Methods
The core contribution is the concept of "story engineering," a process for transforming fictional characters into SSCs. This involves three key steps:
- Character and Story Creation: Defining the SC's personality, worldview, and background narrative.
- Presenting Live Stories to the Community: The SC recounts daily challenges and seeks suggestions from the community, creating an ongoing narrative.
- Communication with Community Members: Enabling direct interaction between users and the SC, allowing for discussion and influence on the story's progression.
The authors employed GPT-3 to drive their SSC prototypes, "David" and "Catherine," and developed a system that includes a "Clue Finder" for static replies and an LLM for dynamic responses. The system also incorporates a word filter for safety and a dialogue history to maintain conversational coherence.
Datasets
The study was conducted within the "DE (Alias)" online gaming community on Discord. The community already had a non-storytelling LLM-based chatbot named Jerry, which served as a benchmark for comparison. No new datasets were introduced; rather, existing community interactions and user feedback were collected.
Key Insights and Ideas
- Storytelling as a Believability Enhancer: Fictional narratives and character backstories significantly increase the believability and engagement of social chatbots in community settings.
- Community Integration is Key: SSCs can act as catalysts for community activity, driving engagement even when overall activity is low.
- Emotional Depth Drives Connection: Characters that express emotions and demonstrate distinct personalities foster stronger connections with users.
- Autonomy and Disobedience Spark Curiosity: Allowing characters to exhibit autonomy, even through disobedience, can pique user interest and enhance their perceived personality.
- LLM Improvisation as a Creative Tool: LLM-generated content, even if fabricated, can unexpectedly inspire story development and lead to more engaging narratives.
- Beyond Dyadic Interactions: Community-based SSCs facilitate richer social experiences, including indirect connections between users, which are not possible in one-on-one chatbot interactions.
Key Results
- Quantitative: Catherine, the SSC, consistently scored higher than the benchmark chatbot Jerry in believability metrics. While Jerry had slightly higher scores in consistency, satisfaction, and usefulness, Catherine excelled in emotion and engagement. Catherine's chat channel accounted for 89% of the total community activity during her event days.
- Qualitative: Community members reported feeling more emotionally connected to Catherine, describing her as more human-like and engaging due to her story and evolving responses. Players found the daily story updates and interactive choices fun and intriguing. Catherine's role as a community chatbot fostered more diverse and engaging social interactions.
Limitations and Future Work
The authors acknowledge several limitations:
- Small Sample Size: The study involved a limited number of participants, which may affect the generalizability of the findings.
- Gender Imbalance: The participant demographics reflected the underrepresentation of females in Web3 communities, suggesting a need for future research to explore gender-balanced communities.
- Fixed Responses: While necessary for control, Catherine's reliance on fixed responses and story control sometimes led to repetition and logical inconsistencies.
- LLM Evolution: The study was conducted before the widespread availability of advanced LLMs like ChatGPT, suggesting that future SSCs could demonstrate even greater performance.
Future work includes developing more systematic and adaptable approaches for community storytelling, exploring LLMs like ChatGPT and GPT-4 for more flexible story generation, and revisiting findings in light of advancing AI technology. The authors also suggest exploring more theoretical and practical approaches to interactions with narrative-driven AI characters, applicable to both chatbots and game characters.
Summary updated