Bardo: Emotion-Based Music Recommendation for Tabletop Role-Playing Games

Rafael Rodrigues Padovani, Lucas N. Ferreira, Levi H. S. LelisAAAI 2021 · DOI

Abstract

In this paper, we introduce Bardo, a real-time intelligent system designed to automatically select background music for tabletop role-playing games (TTRPGs). Bardo utilizes off-the-shelf speech recognition to transcribe player dialogue and a supervised learning algorithm to classify the emotional state of the scene into one of four categories: Happy, Calm, Agitated, or Suspenseful. Based on this classification, the system dynamically selects and plays appropriate background music. Evaluation using a Dungeons & Dragons (D&D) campaign demonstrates that a Naive Bayes classifier with a sliding window approach achieves high prediction accuracy. Furthermore, a user study indicates that Bardo’s automated selections are often preferred over the original human-curated music.

Introduction: The AI Dungeon Master’s Personal DJ

In TTRPGs like Dungeons & Dragons, immersion is a primary goal. While background music significantly enhances the atmosphere, managing it manually is a significant burden for Dungeon Masters (DMs), who must already manage complex rules, non-player characters, and narrative flow. Automating this process is challenging due to the "messy" nature of live play, characterized by ungrammatical speech, overlapping dialogue, and rapid emotional shifts. Bardo addresses these challenges by providing a real-time, context-aware music recommendation system.

Methodology

The Bardo pipeline consists of three main stages:

  1. Speech-to-Text: The system captures player dialogue using an off-the-shelf speech recognition system. To maintain robustness against transcription errors, the text is processed as a "bag of words," prioritizing keywords over complex syntax.
  2. Emotion Classification: We compared lexicon-based approaches (Density-Based) with probabilistic models (Naive Bayes). The system classifies dialogue into four emotional states: Happy, Calm, Agitated, and Suspenseful.
  3. FSM Controller: To ensure smooth musical transitions and prevent rapid "jittering," Bardo employs a Finite State Machine (FSM). This controller uses a sliding window to incorporate temporal context and a confidence threshold to trigger music changes.

Dataset and Annotation

The system was trained and evaluated on a custom dataset derived from the YouTube D&D campaign Call of the Wild. The dataset comprises 5,892 sentences (approximately 4.5 hours of gameplay) labeled by three human annotators. This dataset highlights the unique linguistic patterns of TTRPG players, where specific "game-talk" (e.g., "initiative," "stealth") serves as a strong indicator of emotional state.

Key Results

Classification Accuracy

Experimental results showed that the Naive Bayes variant utilizing a sliding window (NS) was the most effective model. It achieved an average accuracy of 64% across nine episodes, which is significant given the inherent noise in transcribed live speech.

Accuracy results across nine episodes for various classification algorithms. The Naive Bayes with sliding window (NS) achieved the highest average accuracy.

User Preference

A user study involving 61 participants compared Bardo’s music selections against the original human-edited audio from the Call of the Wild videos. Participants preferred Bardo’s selections 163 times, compared to 74 for the baseline. Notably, in some instances, Bardo’s automated choices were deemed more appropriate than the original human labels upon retrospective review.

User preference percentages for Bardo versus the original video baseline across five excerpts.

Discussion and Limitations

While Bardo demonstrates the feasibility of automated music selection for TTRPGs, several limitations remain. The "Happy" emotion was underrepresented in the training data, making it difficult for the model to learn. Additionally, the current system relies solely on text, ignoring acoustic features like tone and prosody. Future work will focus on integrating multi-modal data and testing the system in live, face-to-face environments with higher ambient noise and overlapping speech.

Summary updated

Abstract

In this paper we introduce Bardo, a real-time intelligent system to automatically select the background music for tabletop role-playing games. Bardo uses an off-the-shelf speech recognition system to transform into text what the players say during a game session, and a supervised learning algorithm to classify the text into an emotion. Bardo then selects and plays as background music a song representing the classified emotion. We evaluate Bardo with a Dungeons and Dragons (D&D) campaign available on YouTube. Accuracy experiments show that a simple Naive Bayes classifier is able to obtain good prediction accuracy in our classification task. A user study in which people evaluated edited versions of the D&D videos suggests that Bardo's selections can be better than those used in the original videos of the campaign.
BibTeX
@inproceedings{padovani2021bardo,
  author = {Rafael Rodrigues Padovani and Lucas N. Ferreira and Levi H. S. Lelis},
  title = {{Bardo: Emotion-Based Music Recommendation for Tabletop Role-Playing Games}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aiide.v13i1.12958},
}
Figure