TL;DR
This paper systematically evaluates GPT-3's ability to estimate the Big 5 personality traits from social media posts in a zero-shot setting. While GPT-3 shows promise when provided with trait definitions or survey item descriptions, its performance significantly drops when asked for fine-grained classifications, suggesting LLMs are not yet ready for nuanced personality assessment without extensive fine-tuning.
What Problem Does This Paper Address?
Understanding human personality through language is a long-standing challenge in computational social science. While large language models (LLMs) like GPT-3 have demonstrated impressive zero-shot capabilities across various NLP tasks, their effectiveness in understanding complex psychological concepts like personality traits remains largely unexplored. This paper aims to fill this gap by rigorously testing GPT-3's zero-shot performance on estimating the Big 5 personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) from user-generated text, specifically Facebook posts. The authors are interested in how much personality information is implicitly encoded in LLMs and how this compares to existing state-of-the-art (SotA) methods.
Methods
The core of this research involves designing and evaluating different prompting strategies for GPT-3. The authors start with a "BASIC" prompt that simply asks GPT-3 to classify a user's personality trait based on their posts. They then explore the impact of injecting external knowledge into the prompt, testing three types:
- TEXTBOOK: Concise definitions of each personality trait.
- WORDLIST: Lists of words frequently associated with high and low scores for each trait.
- ITEMDESC: Survey items (questions) used in personality questionnaires, presented as positive and negative examples.
GPT-3's outputs were restricted to single tokens for binary classification ("yes" or "no") and three tokens for a three-class classification ("low," "medium," "high"). The performance was evaluated using Macro F1 scores, a metric that averages the F1 score across all classes, giving equal weight to each. They also compare GPT-3's performance against a strong baseline, WT-LEX, a ridge regression model trained on n-grams and LDA topics.
Datasets
The study utilizes a dataset collected by Jose et al. (2022), which comprises Facebook posts from 202 participants who also completed the Big 5 personality test. To ensure data quality and comparability, the authors filtered the data to include only users with exactly 20 posts from the last year of data collection. Personable identifiable information was anonymized using SciPy's NER model. The final dataset consists of anonymized Facebook posts from 142 users, along with their Big 5 personality trait scores, which range from 1 to 5 and were subsequently discretized into classes for classification.
Key Insights and Ideas
- Knowledge Injection Matters: Simply asking GPT-3 to assess personality based on posts yields mediocre results. However, providing it with explicit information about the traits (definitions, word lists, or survey items) significantly boosts its performance.
- Survey Items are Powerful: The ITEMDESC prompt, which uses descriptions of survey items, consistently yielded the best results for GPT-3, particularly for Extraversion and Conscientiousness. This suggests that LLMs can leverage concrete descriptions of behaviors and attitudes to infer personality.
- Lexical Cues are Helpful for Some Traits: For Agreeableness, Openness, and Neuroticism, the WORDLIST prompt performed best. This indicates that for certain traits, simple lexical associations are more effective for GPT-3 than abstract definitions or survey item descriptions.
- Neuroticism is Tough: Estimating Neuroticism proved to be difficult for GPT-3, mirroring human challenges in zero-acquaintance contexts. The model also showed a bias towards predicting high openness and low neuroticism across the board.
- Fine-Grained Classification is a Hurdle: GPT-3's performance drops dramatically when asked to classify traits into three levels (low, medium, high) compared to binary classification. This suggests that LLMs struggle with nuanced, multi-class personality estimation in a zero-shot setting.
Key Results
- ITEMDESC Prompting: Achieved the best overall Macro F1 score for GPT-3, outperforming the BASIC prompt and other knowledge injection methods.
- Comparison to WT-LEX: While GPT-3 with the ITEMDESC prompt showed improvement over the MFC (Most Frequent Class) baseline, its average performance was still lower than the established WT-LEX model.
- Three-Class vs. Binary: The Macro F1 score dropped significantly from 0.454 (binary) to 0.230 (three-class) when increasing the classification granularity, highlighting GPT-3's limitations in fine-grained estimation.
- Trait-Specific Performance: GPT-3 performed best on Extraversion and Conscientiousness when using the ITEMDESC prompt, and on Agreeableness, Openness, and Neuroticism when using the WORDLIST prompt.
Limitations and Future Work
The authors acknowledge several limitations:
- Binary vs. Continuous: The Big 5 model is inherently continuous, and forcing it into a binary or even three-class classification simplifies the problem, potentially masking GPT-3's true capabilities or limitations in regression tasks.
- Prompt Sensitivity: While they explored different knowledge types, the exact composition and ordering of knowledge in prompts could further influence performance, and this was left for future work.
- Cost and Context Window: The significant cost of using the GPT-3 API and its limited context window restricted the extent of in-context learning experiments.
Future work could focus on:
- Exploring regression-based approaches for personality estimation with LLMs.
- Investigating more sophisticated prompt engineering techniques, including combining different knowledge types.
- Understanding the impact of larger context windows and more extensive in-context learning on GPT-3's personality estimation abilities.
- Further analyzing the linguistic features that contribute to GPT-3's errors and successes to guide improvements in LLM capabilities for human-level NLP tasks.
Summary updated