TL;DR
This paper proposes a novel approach to personality profiling on social media by using items from validated psychometric tests as training data for BERT classifiers. This method bypasses the need for costly, manually annotated social media data, achieving comparable performance to in-domain training for most personality traits, especially when combined with T5-based data augmentation.
What Problem Does This Paper Address?
Automatically profiling users on social media for traits like personality is a valuable task, offering insights into demographics, psychological states, and more. Traditional methods often rely on expensive data collection strategies: either asking users to complete psychometric tests and share their social media profiles, or inferring labels from user-provided information (which can be noisy or biased). Expert annotation is also an option but is labor-intensive. This paper addresses the challenge of acquiring high-quality, labeled social media data for personality profiling by exploring a more cost-effective alternative: directly leveraging the existing, validated textual items from psychometric tests as training data. This approach aims to overcome the limitations of small, specialized datasets while still achieving robust profiling capabilities.
Methods
The core of the paper's methodology involves using items from psychometric tests as training data for BERT classifiers. The workflow is as follows:
- Corpus Creation: All items from the IPIP-NEO-300 psychometric test (measuring the Big Five personality traits: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) are collected. Each item is assigned a binary label (positive/negative correlation) based on the test's evaluation guidelines.
- Data Augmentation: To combat the "small data" problem inherent in using psychometric test items, three data augmentation techniques are employed:
- Easy Data Augmentation (EDA): Utilizes synonym replacement, random insertion, deletion, and swap operations.
- T5 Item Paraphrasing: Leverages the T5 model to generate paraphrased versions of the test items.
- GPT-2 Item Generation: Uses GPT-2 to generate new items for each personality trait.
- Classification Model: BERT (specifically
bert-base-uncased) is fine-tuned on the augmented psychometric test item data. For each personality trait, a separate BERT model is trained. - Inference and Aggregation: The fine-tuned BERT models are used to predict personality trait labels for individual tweets from a Twitter evaluation corpus. These individual tweet predictions are then aggregated (using majority voting) to determine the overall profile label for each user.
- Evaluation: The predicted profile labels are compared against the "gold standard" annotated labels from the PAN-author-profiling-2015 dataset. A comparison is also made against an "in-domain" model trained directly on Twitter data.
Datasets
- Training Data: Items from the IPIP-NEO-300 psychometric test, covering the Big Five personality traits. This corpus is augmented using EDA, T5 paraphrasing, and GPT-2 generation.
- Evaluation Data: The English subset of the PAN-author-profiling-2015 dataset, which consists of Twitter profiles annotated with personality traits.
Key Insights and Ideas
- Psychometric Items as Proxies: The paper demonstrates that validated psychometric test items, despite being "small data," can serve as effective training material for social media personality profiling models, bypassing the need for direct social media annotations.
- T5 Augmentation's Strength: T5-based paraphrasing emerged as a particularly effective data augmentation strategy, significantly improving performance for several personality traits and often matching or exceeding in-domain training results.
- Transferability Across Domains: The success of fine-tuning on psychometric items and then applying to Twitter data highlights the generalizability of language models and the semantic richness captured by psychometric instruments.
- Trait-Specific Performance: While the approach shows promise across most traits, performance varies, with Neuroticism being an exception where plain psychometric models perform comparably to in-domain models, and Conscientiousness showing substantially higher performance than Neuroticism.
Key Results
- The proposed method, using psychometric test items with T5 data augmentation, achieved performance comparable to in-domain training for 4 out of 5 personality traits (Conscientiousness, Extraversion, Agreeableness, and Neuroticism).
- For Conscientiousness, Extraversion, and Agreeableness, T5 augmentation led to substantial improvements, with F1 scores reaching .89, .82, and .65 respectively, outperforming the plain psychometric models and sometimes even the in-domain models.
- The "plain" psychometric models (without augmentation) were generally outperformed by a random baseline, except for Neuroticism, where they performed on par with the in-domain model.
- The in-domain model served as a strong baseline, but the psychometric-item-based approach with augmentation closed the gap significantly.
Limitations and Future Work
- Limited Test Instances: The paper acknowledges that the "small data" nature of psychometric test items is a significant challenge, which they addressed with data augmentation.
- Trait-Specific Challenges: The performance varied across personality traits, indicating that some traits might be more challenging to profile using this method or that the psychometric items themselves might not fully capture the nuances of social media expression for certain traits. Openness, in particular, had fewer test instances and showed less improvement.
- Future Work Suggestions:
- Developing models that consistently perform well across all concepts, potentially by combining augmentation methods.
- Exploring zero-shot learning setups where items are used as prompts.
- Investigating the use of these models for more sensitive concepts, like mental health, with careful consideration for privacy and potential misuse.
- Ensuring that such systems are deployed in a way that prevents personalized results to mitigate misuse.
Summary updated