Abstract
Automatic personality recognition is a critical task in computational psycholinguistics, traditionally focused on static monologue datasets. This paper introduces the FriendsPersona dataset, a novel resource for multi-party dialogue analysis derived from the TV show Friends. By leveraging modern attentive neural networks and contextual embeddings like BERT and RoBERTa, the researchers establish new state-of-the-art (SOTA) benchmarks. Results demonstrate a 2.49% average accuracy improvement on the classic Essays dataset and provide a robust baseline for personality detection in complex conversational contexts.
Introduction
Understanding a person's character through their language involves predicting the "Big Five" personality traits: Agreeableness (AGR), Conscientiousness (CON), Extraversion (EXT), Openness (OPN), and Neuroticism (NEU). Historically, this field has relied on traditional linguistic features and the "Essays" dataset—a collection of stream-of-consciousness monologues from the late 1990s. However, real-world personality is frequently expressed through social interaction. This research addresses the lack of dialogue-based datasets and the reliance on outdated models by introducing a new dataset and applying state-of-the-art Transformer architectures.
Datasets and Methodology
Datasets
The study utilizes two primary benchmarks:
- Essays Dataset: A classic benchmark containing 2,468 self-report essays.
- FriendsPersona: A novel dataset consisting of 711 conversations from the first four seasons of Friends. Each segment was annotated for the Big Five traits by three independent workers.
The MainSpeakerFinder (MSF) Algorithm
To extract coherent dialogue segments, the authors developed the MSF algorithm. It identifies "sub-scenes" where a specific character is the primary speaker by using a sliding window to track utterance counts and identifying peaks in conversation flow. This ensures the extracted text is speaker-centric rather than a random snippet.
Modeling Suite
The researchers compared several architectures to evaluate their effectiveness in personality recognition:
- ABCNN & ABLSTM: Convolutional and LSTM networks enhanced with attention mechanisms.
- HAN (Hierarchical Attention Network): A model designed to capture hierarchies by looking at both word-level and utterance-level features.
- Contextual Embeddings (BERT & RoBERTa): Fine-tuned Transformer models that capture deep semantic nuances based on surrounding context.
Results and Analysis
Performance on Monologues (Essays)
The RoBERTa-based model achieved a new SOTA on the Essays dataset, outperforming previous benchmarks by an average of 2.49% across all five traits. As shown in the results table, RoBERTa dominated four out of five categories, while ABCNN showed surprising strength in predicting Conscientiousness (CON).
Performance on Dialogues (FriendsPersona)
On the new FriendsPersona dataset, models were tested using three formats: Speaker-only (S), Speaker + Context (S+C), and Full dialogue (F). Interestingly, models generally performed best in the "S" format. This suggests that current general-purpose models like BERT may struggle to filter out the "noise" of interlocutors when trying to isolate a single speaker's personality.
While RoBERTa achieved the highest accuracy for Agreeableness (65.58%) and Openness (68.47%), the Hierarchical Attention Network (HAN) proved particularly effective for dialogue. By explicitly modeling the relationship between tokens and utterances, HAN is better equipped to identify specific conversational moments that reveal character traits.
Discussion and Future Work
A significant challenge noted by the researchers was the low Fleiss’ kappa among human annotators, highlighting the inherent subjectivity of personality recognition. To improve performance, the authors suggest moving toward multimodal data, incorporating audio and video cues to supplement text. Future work will also focus on developing customized BERT-based attention networks specifically designed to model inter-speaker relationships in multi-party dialogues.
Summary updated