Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention

Veronica E. Lynn, Niranjan Balasubramanian, H. Andrew SchwartzACL 2020 · DOI

Abstract

Predicting user personality from social media text often suffers from a low signal-to-noise ratio, as mundane daily updates can dilute psychologically informative content. This paper presents a hierarchical neural architecture that utilizes dual-level attention—at both the word and message levels—to identify and prioritize high-signal messages. Evaluated on a large-scale dataset of Facebook users, the model achieves state-of-the-art performance across the "Big Five" personality traits, demonstrating that selective attention to specific messages is crucial for accurate language-based personality assessment.

Introduction: The Challenge of Signal vs. Noise

Language-based personality assessment aims to infer psychological traits from digital footprints. Traditional approaches often treat a user's entire corpus as a single document or aggregate features across all posts equally. However, human communication is inconsistent; only a fraction of a user's posts may contain relevant psychological cues. For example, a post about a social gathering is highly indicative of extraversion, whereas a mundane update about a technical issue acts as noise.

Examples of high-signal vs. low-signal messages identified by the model.

Existing models that average embeddings or use "bag-of-words" representations often "wash out" these rare, highly descriptive moments. To address this, we propose a model capable of automatically distinguishing between high-signal and low-signal messages to improve prediction accuracy.

Hierarchical Dual-Attention Architecture

The core of our approach is a Hierarchical Sequence Model featuring a dual-attention mechanism designed to capture importance at two granularities:

  1. Word-to-Message Level: Each individual message is processed by a Gated Recurrent Unit (GRU). A word-level attention mechanism identifies which words within that specific message are most salient (e.g., "party" vs. "the"). This produces a fixed-length vector representation for each message.
  2. Message-to-User Level: The sequence of message vectors is then processed by a second GRU layer. A message-level attention mechanism assigns a weight to each message, effectively determining its contribution to the final user-level personality profile.
The hierarchical neural model architecture featuring word-level and message-level attention.

The resulting user representation is passed through fully connected layers to predict scores for the "Big Five" traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (OCEAN).

Dataset and Methodology

The study utilized a massive dataset of Facebook status updates from 68,687 consenting users, collected via the myPersonality application.

  • Data Filtering: To ensure robust signal, users were required to have authored at least 1,000 words.
  • Ground Truth: Personality labels were derived from validated psychological questionnaires ranging from 20 to 100 items.
  • Evaluation: A high-quality subset of 1,943 users who completed the full 100-item questionnaire served as the "gold standard" test set.

Key Insights and Performance

The hierarchical model achieved state-of-the-art results, with an average increase in Pearson r of 2.5% over previous benchmarks. Several key findings emerged:

  • Primacy of Message-Level Attention: Ablation studies showed that while word-level attention is beneficial, message-level attention provides the most significant performance boost. It prevents informative signals from being diluted by the sheer volume of mundane posts.
  • Sparsity of Signal: For traits like Agreeableness, the model found that high-signal messages are rare but extremely predictive. Performance often plateaus after identifying a few key "hits," suggesting that a user's entire history may not be necessary if the right snippets are identified.
  • Hybrid Modeling: The highest accuracies were achieved by combining the neural model's predictions with traditional feature-based models (n-grams and LDA topics). This suggests that neural architectures and traditional linguistic features capture complementary information.
  • Expert Validation: In a blind evaluation, personality experts agreed with the model's "high-attention" selections 75% of the time for traits like Openness and Agreeableness, confirming the model's alignment with human psychological intuition.

Conclusion

This research highlights the importance of hierarchical modeling in user-level tasks. By explicitly modeling the relative importance of different messages, we can better navigate the noise inherent in social media data. Future work will focus on scaling these models to longer user histories and exploring their cross-platform and cross-cultural generalizability.

Summary updated

BibTeX
@inproceedings{lynn2020hierarchical,
  author = {Veronica E. Lynn and Niranjan Balasubramanian and H. Andrew Schwartz},
  title = {{Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.472},
}
Figure