Abstract
Expressing attitude or stance toward entities and concepts is an integral part of human behavior and personality. Recently, evaluative language data has become more accessible with social media's rapid growth, enabling large-scale opinion analysis. However, surprisingly little research examines the relationship between personality and evaluative language. To bridge this gap, we introduce the notion of evaluative topics, obtained by applying topic models to pre-filtered evaluative text from social media. We then link evaluative topics to individual text authors to build their evaluative profiles. We apply evaluative profiling to Reddit comments labeled with personality scores and conduct an exploratory study on the relationship between evaluative topics and Big Five personality facets, aiming for a more interpretable, facet-level analysis. Finally, we validate our approach by observing correlations consistent with prior research in personality psychology.
Introduction
While we have long understood that language reflects personality, most automated personality analysis treats all text equally. However, a person describing a mundane object provides less psychological "signal" than a person explaining why they hate a specific political policy or why they love a certain genre of music. This paper addresses the "signal-to-noise" problem of social media, where most text is non-evaluative, by focusing specifically on evaluative language—the stances and opinions individuals express.
By building "evaluative profiles," the researchers demonstrate that our stances on topics like music, fiction, and social issues are much stronger predictors of our personality facets than general, non-opinionated chatter. Furthermore, the study moves beyond broad "Big Five" traits (like Openness) to map language to more granular personality facets (like "artistic interests" or "assertiveness"), providing a higher-resolution psychological profile.
Methodology
The researchers developed a three-step pipeline to transform raw social media text into a psychological profile:
- Evaluative Filtering: To isolate opinions, the team used sentiment analysis (VADER), opinion lexicons, and a novel technique called Quasi-Snowballing (QSB). QSB is an iterative paraphrase mining technique that uses sentence transformers to find similar opinionated statements, significantly increasing the volume of usable data.
- Evaluative Topic Induction: Various topic models were applied to the filtered text. The Combined Topic Model (CTM)—which blends variational autoencoders with BERT-based embeddings—performed the best, identifying 20 distinct "evaluative topics" such as investing/finance, fiction, and maltreatment.
- Evaluative Profiling & Analysis: User profiles were constructed based on topic prevalence and sentiment intensity. Finally, Canonical Correlation Analysis (CCA) was used to identify the strongest relationships between these topic profiles and the users' self-reported personality scores from the PANDORA dataset (1.3 million Reddit comments from 1,608 users).
Key Results and Insights
The core finding of the study is the "Evaluative Advantage": evaluative text contains significantly more information about personality than non-evaluative text. When models were run on "non-evaluative" text (the leftovers from the filter), correlations with personality traits dropped sharply.
Other key insights include:
- Brunswik Symmetry Principle: Meaningful patterns emerge when the "granularity" of the data matches. Specific facets (like "artistic interests") align more naturally with specific evaluative topics (like "music") than broad traits do.
- Sentiment Intensity: Including the intensity of the sentiment (how strongly someone feels) improved the correlation with personality traits compared to looking at topic prevalence alone.
- Psychological Clusters: The analysis revealed distinct clusters, such as an Openness Cluster (linking music/fiction to imagination) and a Social Engagement Cluster (linking politics/social issues to assertiveness).
The study found numerous correlations exceeding .40 between specific topic-facet pairs, which is notably higher than the average correlation typically found in traditional questionnaire-based personality studies (~.19).
Discussion and Future Work
While the results are promising and consistent with established psychological literature, the study notes some limitations. The high-resolution "facet-level" data was limited to a subset of 127 users, and the current pipeline is English-centric. Future work could combine implicit and explicit sentiment polarity to further refine evaluative profiles, with potential applications in mental health prediction and personalized recommender systems.
Summary updated