TL;DR
This paper introduces a novel method for recognizing apparent personality traits from multimodal human behavior by jointly modeling both the overall Big Five personality scores and the finer-grained responses to individual questionnaire items. By leveraging this richer information, the authors demonstrate improved personality trait recognition and introduce a new dataset to facilitate this research.
What Problem Does This Paper Address?
Recognizing personality traits from human behavior, particularly "apparent personality" (how others perceive us), is a growing area of interest. The "Big Five" model (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) is a standard framework. Traditionally, this recognition relies on analyzing multimodal cues like speech and visual expressions. However, the ground truth for training these models is often derived from personality questionnaires. While these questionnaires provide scores for the Big Five traits, they also contain detailed responses to individual items (e.g., "Is the life of the party"). Previous research has largely ignored this granular item-level information, treating it solely as a means to calculate the Big Five scores. This paper argues that this item-level data contains valuable "meta-level" information about personality characteristics that can enhance the overall personality recognition task and improve our understanding of multimodal human behavior.
Methods
The core of the proposed method is a multimodal transformer architecture that acts as a backbone for processing audio, visual, and text (derived from speech via ASR) inputs. This architecture allows for effective cross-modal interaction. The novelty lies in the joint modeling of Big Five scores and questionnaire item-level scores. Two joint modeling approaches are presented:
- Multi-Task Joint Model: This model uses two separate prediction heads attached to the multimodal transformer backbone. One head predicts the Big Five scores, and the other predicts the questionnaire item-level scores. Both are trained simultaneously.
- Cascaded Joint Model: This approach first predicts the questionnaire item-level scores and then uses these predictions as input to predict the Big Five scores.
Both models are trained using a loss function that combines the mean absolute error for both Big Five scores and questionnaire item-level scores, with a hyperparameter to balance their contributions. The authors also explore the impact of pre-training modal-specific encoders (audio, text, visual) on tasks like masked prediction, masked language modeling, and face recognition/expression recognition, respectively.
Datasets
The paper introduces a newly created self-introduction video dataset. This dataset comprises:
- 10,100 self-introduction videos from 1,010 Japanese participants.
- Each video is associated with annotations from a 50-item Big Five questionnaire.
- The questionnaire responses were collected from multiple assessors (5 for training/validation, 10 for testing) to ensure robust ground truth.
This dataset is significant because existing popular datasets for personality recognition (e.g., ChaLearn first impression, UDIVA) do not provide this granular questionnaire item-level data, limiting previous research.
Key Insights and Ideas
- Item-level data is valuable: The paper strongly suggests that the detailed responses to individual questionnaire items, beyond just the aggregated Big Five scores, contain rich information that can significantly improve personality trait recognition.
- Joint modeling enhances performance: By jointly predicting both Big Five scores and item-level scores, the model gains a better understanding of multimodal human behavior, leading to improved Big Five score estimation.
- Multi-task learning is effective: The multi-task joint model, which predicts both types of scores simultaneously, generally outperformed the cascaded model and single-task models for Big Five estimation.
- Pre-training is crucial: Pre-training the individual modal encoders (audio, text, visual) on relevant tasks significantly boosts performance across all modalities.
- More items, better awareness: Increasing the number of questionnaire items included in the joint modeling task led to improved performance, suggesting that richer item-level information directly translates to better multimodal behavior awareness.
Key Results
- The proposed multi-task joint model significantly outperformed the single-task model in estimating Big Five scores.
- The multi-task joint model achieved performance comparable to or exceeding human evaluation for Big Five score estimation.
- The model demonstrated effective estimation of questionnaire item-level scores, with performance also exceeding human evaluation in some aspects.
- Using all three modalities (audio, text, visual) with pre-trained encoders yielded the best results.
- Performance improved as the number of questionnaire items used in the joint model increased, with the 50-item questionnaire showing the best results.
Limitations and Future Work
- Dataset limitations: While a new dataset is introduced, it consists of self-introduction videos from Japanese participants. Future work could explore datasets from diverse cultural backgrounds and more naturalistic interaction scenarios.
- Model complexity: The multimodal transformer architecture, while effective, can be computationally intensive.
- Cascaded model performance: The cascaded joint model did not consistently outperform the single-task model, suggesting that the direct use of item-level predictions might not always be beneficial for Big Five estimation compared to indirect consideration.
- Future directions: The authors suggest exploring more sophisticated joint modeling techniques, investigating the interpretability of the learned representations, and applying the method to other personality assessment frameworks.
Summary updated