Multi-Document Transformer for Personality Detection

Feifan Yang, Xiaojun Quan, Yunyi Yang, Jianxing YuAAAI 2021 · DOI

Abstract

Detecting personality traits from social media is challenging because relevant signals are often scattered across hundreds of independent posts. Traditional models either treat posts in isolation or concatenate them into a single sequence, which introduces an arbitrary "post-order bias." This paper introduces Transformer-MD, a multi-document framework that treats a user's post history as an unordered set. By employing a two-stage encoding process and a novel Dimension Attention mechanism, the model effectively captures inter-post interactions and links specific posts to distinct personality dimensions, achieving state-of-the-art results on MBTI benchmarks.

Introduction

Personality detection, particularly for the Myers-Briggs Type Indicator (MBTI), aims to identify psychological traits from digital footprints. Social media data presents two primary hurdles:

  1. Interaction vs. Independence: Encoding posts individually misses holistic connections, while concatenation introduces "post-order bias"—the model learns spurious patterns from the chronological sequence of posts.
  2. Trait-Specific Signals: Personality is multidimensional. A single post might be highly relevant for "Introversion vs. Extroversion" but irrelevant for "Thinking vs. Feeling." Existing models often fail to distinguish these trait-specific signals.

Methodology: Transformer-MD

The proposed Transformer-MD (Multi-Document) architecture addresses these challenges through a hierarchical approach and order-agnostic fusion.

The architecture of Transformer-MD, featuring low-level encoding and multi-document fusion.

Two-Stage Encoding

To manage the high token count of extensive post histories, the model utilizes a Low-Level Encoder to process each post individually. This stage aggregates information into a [CLS] token for every post.

Post-Order-Agnostic Fusion

The resulting [CLS] tokens are fed into a High-Level Encoder. To ensure the model treats posts as an unordered set, all tokens share the same position embedding. This mathematical constraint prevents the model from being biased by the temporal order of posts.

Dimension Attention (DA)

A trait-specific attention layer is implemented on top of the encoder. Instead of a single user vector, the model generates four distinct representations—one for each MBTI dimension—by selectively attending to the most relevant post information for that specific trait.

Experimental Setup

The model was evaluated on two MBTI benchmarks:

  • Kaggle MBTI: ~8,000 users with 40–50 posts each.
  • Pandora MBTI: A larger Reddit-based dataset with hundreds of posts per user. Macro-F1 was used as the primary metric to account for the inherent class imbalance in personality distributions.

Results and Analysis

The "Order Bias" Trap

The authors demonstrated that standard models like BERT and XLNet overfit to the arbitrary order of posts. As shown in the figure below, when post order is shuffled at test time, the performance of these standard models drops significantly, whereas Transformer-MD remains stable.

Performance drop in standard models when post order is randomized, illustrating post-order bias.

Performance and Ablation

Transformer-MD achieved state-of-the-art results, reaching 70.47% Macro-F1 on Kaggle (a 4.61% improvement over previous SOTA) and 60.92% on Pandora. Ablation studies revealed that the Multi-Document Fusion Layer—which facilitates inter-post interaction—is the most critical component for accurate detection.

Analysis of attention weights showed that the top 20% of inter-post links provide the most predictive power, often connecting posts with similar emotional tones. Furthermore, the Dimension Attention module performs better when utilizing word-level information, suggesting that specific keywords are vital indicators for certain traits.

Conclusion

Transformer-MD effectively mitigates post-order bias while capturing the complex, multidimensional nature of personality. While computationally intensive, its success on "Introversion vs. Extroversion" and "Thinking vs. Feeling" dimensions marks a significant advancement. Future work may focus on improving performance for more subtle dimensions like "Sensing vs. Intuition" and adapting the architecture for other multi-document tasks where order is irrelevant.

Summary updated

BibTeX
@inproceedings{yang2021multi,
  author = {Feifan Yang and Xiaojun Quan and Yunyi Yang and Jianxing Yu},
  title = {{Multi-Document Transformer for Personality Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17673},
}
Figure