Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset

Bin Tang, Keqi Pan, Miao Zheng, Ning Zhou, Jialu Sui, Dandan Zhu, Cheng-Long Deng, Shu-Guang KuaiAAAI 2025 · DOI

Introduction

Predicting the "Big Five" personality traits—Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (OCEAN)—is a cornerstone of affective computing. Despite its importance, current AI models for personality recognition face two primary limitations. First, most existing datasets focus on facial expressions, speech, or text, neglecting the critical role of full-body language. Psychological research has long established that traits like extraversion are strongly manifested through broad, expansive gestures. Second, many models employ "naive fusion," treating all input modalities as equally relevant for every trait, whereas psychology suggests that specific traits are better reflected in specific modalities (e.g., Agreeableness in facial cues versus Conscientiousness in posture).

Visualization of the relationship between full-body pose and extraversion levels in the new dataset.

To address these gaps, this research introduces the Psychology-Inspired Network (PINet) and a comprehensive new multimodal dataset that prioritizes high-quality, frontal, full-body pose data as a primary modality.

The PINet Architecture

PINet is designed to move beyond the computational and conceptual limitations of standard Transformer-based architectures. It consists of three core innovations:

  1. Multimodal Feature Awareness (MFA): Utilizing Vision Mamba Blocks, PINet captures subtle spatio-temporal features in video and pose data. Unlike Transformers, which suffer from quadratic computational complexity, the Mamba-based architecture maintains efficiency while modeling long-range dependencies.
  2. Multimodal Feature Interaction (MFI): This module employs Fusion Mamba to integrate five distinct streams: raw frames, cropped faces, full-body pose, audio, and text. This approach allows for complex cross-modal interaction with linear complexity, ensuring the model remains scalable.
  3. Psychology-Informed Modality Correlation Loss (PIMC Loss): This novel loss function acts as the model's "secret sauce." By incorporating psychological priors, PIMC Loss guides the network to prioritize specific modalities for specific traits. For instance, the model is encouraged to focus on pose and text for Conscientiousness, while prioritizing facial and audio features for Agreeableness.

A New Multimodal Dataset

To support this research, the authors developed and released a new benchmark dataset. The data was collected from 287 university students who answered 36 interview questions in a controlled virtual environment.

Experimental setup for the multimodal personality dataset collection.

The dataset provides over 10,000 samples across five streams. Notably, full-body pose data was extracted with high precision using AlphaPose. Unlike previous datasets like SIAP, which often featured limited samples or suboptimal camera angles (e.g., side views), this dataset provides high-precision frontal views, which are essential for capturing the nuances of body language in social interactions. Personality labels were obtained through the validated NEO-FFI-3 self-report scale.

Key Results and Insights

PINet establishes a new state-of-the-art for personality recognition. The model achieved a Pearson Correlation Coefficient (PCC) of 0.2509 and a Concordance Correlation Coefficient (CCC) of 0.2706. This represents a massive improvement over the previous leading model, CRNet, with gains of 119.9% in PCC and 134.2% in CCC.

Ablation studies confirmed the critical role of pose as a modality. Removing pose data resulted in a 36.8% drop in PCC for PINet, demonstrating that body language is a vital "missing link" in personality AI. Furthermore, the study found that pose often outperforms raw video frames and facial crops for certain traits, ranking as the third most important modality overall behind text and audio.

Conclusion and Future Work

This work demonstrates that constraining deep learning models with human psychological theory—specifically regarding modality importance—leads to significantly more accurate and interpretable results. By treating pose as a first-class modality and utilizing efficient Mamba-based architectures, PINet overcomes the bottlenecks of traditional multimodal fusion.

Future research will look to address the limitations of self-report bias by incorporating "observer-reported" labels to capture apparent personality. Additionally, the authors plan to expand the dataset to include more diverse populations and test the model's robustness in less controlled, real-world social environments.

Summary updated

BibTeX
@inproceedings{tang2025pose,
  author = {Bin Tang and Keqi Pan and Miao Zheng and Ning Zhou and Jialu Sui and Dandan Zhu and Cheng-Long Deng and Shu-Guang Kuai},
  title = {{Pose as a Modality: A Psychology-Inspired Network for Personality Recognition with a New Multimodal Dataset}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32145},
}
Figure