Abstract
Expressing attitude or stance toward entities and concepts is an integral part of human behavior and personality. Recently, evaluative language data has become more accessible with social media’s rapid growth, enabling large-scale opinion analysis. However, surprisingly little research examines the relationship between personality and evaluative language. To bridge this gap, we introduce the notion of evaluative topics, obtained by applying topic models to pre-filtered evaluative text from social media. We then link evaluative topics to individual text authors to build their evaluative profiles. We apply evaluative profiling to Reddit comments labeled with personality scores and conduct an exploratory study on the relationship between evaluative topics and Big Five personality facets, aiming for a more interpretable, facet-level analysis. Finally, we validate our approach by observing correlations consistent with prior research in personality psychology.
Introduction: The Signal in the Noise
While language has long been recognized as a mirror of personality, traditional NLP research often focuses on broad word frequencies or general topics. However, personality is fundamentally expressed through our attitudes, values, and stances toward the world. The authors of this study identify a "signal-to-noise" problem: in the massive stream of social media data, factual statements or "noise" often drown out the evaluative "signal" that reveals a person's character.
Furthermore, most studies look at the broad "Big Five" traits (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism), which can be too coarse for nuanced analysis. This paper seeks to bridge the gap by linking specific evaluative topics to granular personality facets (e.g., "artistic interests" instead of just "Openness").
Methods: Building Evaluative Profiles
The researchers proposed a multi-step pipeline to move from raw text to personality profiles:
- Evaluative Filtering & Quasi-Snowballing (QSB): To isolate opinions, they used a combination of sentiment tools (VADER), opinion lexicons, and stance markers. To expand this dataset without losing precision, they developed a "quasi-snowballing" technique. Starting with a small seed of evaluative sentences, they used sentence transformers to iteratively find and add paraphrases from the larger corpus.
- Inducing Evaluative Topics: They tested several topic modeling architectures, including LDA, Biterm Topic Model (BTM), and Attention-based Aspect Extraction (ABAE). The winner was the Combined Topic Model (CTM), which leverages contextual embeddings to produce more coherent topics from short Reddit comments.
- Sentiment-Enhanced Profiling: Instead of just counting how often a user mentions a topic, they constructed "evaluative profiles" where topic prevalence is weighted by sentiment intensity.
- Canonical Correlation Analysis (CCA): Finally, they used CCA to find the shared "latent space" between these evaluative topics and 30 different personality facets.
Datasets
The study utilized PANDORA, a large-scale dataset of Reddit users who have self-reported their personality scores via standardized tests.
- Scale: The total dataset includes 1,608 users with 1.3 million comments (14.3 million sentences).
- Granularity: A subset of 127 users provided high-resolution NEO PI-R facet scores, allowing the researchers to look at 6 specific facets for each of the Big Five traits.
Key Insights
- The "Evaluation" Advantage: Topics induced from filtered evaluative text have a much stronger association with personality than topics induced from raw, unfiltered text. This confirms that our opinions are more "personality-dense" than our factual observations.
- Sentiment as an Amplifier: Including the intensity of the sentiment (how strongly someone feels about a topic) significantly improves the correlation between the text profile and the personality score.
- Natural Clusters: The CCA revealed intuitive "clusters" of traits and topics. For example, "Music" and "Fiction" topics mapped directly to "Artistic Interests" and "Imagination," while "Political" and "Social Issues" clustered with "Assertiveness" and "Activity Level."
Key Results
The researchers found that filtering for evaluative language and including sentiment intensity (Eval+Senti) yielded the highest correlations across canonical dimensions compared to unfiltered or non-evaluative text.
- Model Performance: The CTM model outperformed traditional LDA and BTM in topic coherence (NPMI of 0.0628).
- Correlation Strength: The researchers found several moderate correlations (r > 0.4) between specific topic-facet pairs. This is notably high, considering that the average correlation in traditional questionnaire-based psychology studies is typically around 0.19.
- Validation: The results largely aligned with established psychological literature—for instance, confirming that "Openness" facets correlate positively with curiosity and fiction, and negatively with religious topics.
Limitations and Future Work
The authors acknowledge several hurdles, including the relatively small sample size of users with high-resolution facet scores (n=127) and the limitation to English-speaking Reddit users. Additionally, the current filtering method relies on explicit markers (e.g., "I believe," "IMO") and may miss implicit evaluations.
Ethically, the authors warn against the potential misuse of these techniques for "micro-targeting" or automated decision-making. Future work may involve combining implicit and explicit sentiment analysis and expanding the approach to multi-lingual datasets to see if these "evaluative fingerprints" hold up across different cultures.
Summary updated