TL;DR
This paper explores predicting personality traits from speech, finding that self-supervised speech representations (wav2vec 2.0) significantly outperform traditional expert features. Combining these approaches offers further gains, suggesting that speech alone carries substantial personality information.
What Problem Does This Paper Address?
In the realm of human-computer interaction, virtual assistants that can adapt to a user's personality can lead to more effective and engaging experiences. Automatically inferring personality from speech is a promising avenue for achieving this adaptation. However, accurately predicting personality traits from audio signals remains a challenging task. This paper aims to establish a baseline for apparent personality prediction (APP) using only speech, investigating the effectiveness of different feature extraction methods and modeling techniques.
Methods
The researchers employed two primary modeling approaches:
- Random Forest Regressor (RFR): This model was used to predict all five personality traits simultaneously. It was trained on concatenated sets of features, including:
- Expert Features: Specifically, the eGeMAPS (Geneva Minimalistic Acoustic Parameter Set) which captures a wide range of acoustic and prosodic features commonly used in emotion recognition, and a "speech ratio" (duration of speech vs. total duration).
- wav2vec 2.0 (w2v2) Embeddings: These are powerful, self-supervised representations learned from large amounts of unlabeled speech data. The paper utilized embeddings from all 13 transformer blocks of the w2v2 base model.
- Deep Neural Network (DNN): This model was specifically trained on the sequence of w2v2 embeddings. It was designed to learn a richer representation by combining embeddings from different layers at each time frame before temporal pooling.
The study also explored the impact of background music by comparing performance on subsets of the dataset with and without music. Additionally, they investigated the effect of audio duration on prediction accuracy.
Datasets
The primary dataset used in this study is the "First Impressions" Dataset.
- This is a large, publicly available corpus consisting of 10,000 short (15-second) videos extracted from YouTube vlogs.
- Each video is annotated with five personality ratings corresponding to the Big-five dimensions (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism), collected from external listeners.
- The authors highlight a critical issue with the dataset's official splits, where segments from the same video could appear in both training and testing sets, leading to unrealistically optimistic results. To address this, they created their own data splits, dividing the data by video identifier and stratifying by gender, ethnicity, and average rating.
- For music analysis, they created subsets: "Y-music" (annotated using Yamnet), "M-music" (manually annotated with music), and "M-no_music" (manually annotated without music).
Key Insights and Ideas
- wav2vec 2.0 Dominance: Self-supervised speech representations (wav2vec 2.0) are significantly more effective for apparent personality prediction than traditional expert features like eGeMAPS. This suggests that these learned representations capture deeper, more relevant personality cues.
- Complementary Information: While w2v2 features are powerful, combining them with expert features (like speech ratio) can still yield marginal improvements, indicating that some personality-relevant information might not be fully captured by w2v2 alone.
- The "Speech-Only" Debate: For this dataset, filtering out non-speech segments did not significantly improve performance compared to using the full audio signal, suggesting that the models can learn to ignore irrelevant background noise or that the noise itself might contain subtle cues.
- Splitting Matters: The official data splits of the "First Impressions" dataset lead to inflated performance metrics. Using video-level splits and stratification is crucial for obtaining reliable and generalizable results.
- Agreeableness is Tricky: The "Agreeableness" trait proved to be the most difficult to predict, a finding consistent with previous challenges using this dataset. This might be due to inherent ambiguity or lower inter-annotator agreement for this specific trait.
- Duration is Key: Prediction performance degrades significantly with shorter audio samples, and even at 15 seconds, the performance had not converged, suggesting that longer speech segments could lead to better predictions.
Key Results
- The DNN model trained on w2v2 embeddings achieved the highest performance, outperforming all RFR models.
- The best RFR model using w2v2 embeddings combined with expert features (eGeMAPS + speech ratio) achieved an average R² (Rave) of 0.33 using the official splits, and 0.28 using their proposed, more reliable splits.
- The DNN model achieved a Rave of 0.36 using the official splits and 0.32 using their proposed splits.
- The study found that energy, pitch, and spectral cues (from eGeMAPS) were important for predicting specific traits, with energy being particularly relevant for agreeableness and neuroticism.
- The speech ratio was consistently found to be a useful addition to feature sets.
Limitations and Future Work
- DNN Hyperparameter Tuning: The authors were unable to achieve further gains by combining eGeMAPS with w2v2 embeddings in the DNN model. They attribute this to the need for extensive hyperparameter tuning, which requires holding out test data to avoid optimistic results. This is identified as a key area for future work.
- Music Annotation: While manual annotations for music were performed, their full impact on the task was not deeply explored in this paper, leaving room for further investigation.
- Annotator Agreement: The lack of annotator agreement data for the "First Impressions" dataset makes it difficult to definitively explain why certain traits (like agreeableness) are harder to predict.
- Larger Audio Durations: The performance curve had not converged at 15 seconds, suggesting that exploring longer audio durations could lead to improved prediction accuracy.
- Feature Importance Analysis: While expert feature importance was analyzed, a similar detailed analysis for w2v2 embeddings was not performed.
Summary updated