Abstract
In the Workshop on Computational Personality Recognition (Shared Task), we released two datasets, varying in size and genre, annotated with gold standard personality labels. This allowed participants to evaluate features and learning techniques, and even to compare the performances of their systems for personality recognition on a common benchmark. We had 8 participants to the task. In this paper we discuss the results and compare them to previous literature.
Introduction
Predicting personality from digital footprints—such as social media posts or stream-of-consciousness essays—is a rapidly expanding field with applications in recommendation systems and mental health monitoring. However, the lack of standardized benchmarks has historically hindered progress. Researchers often utilized disparate datasets, evaluation metrics (e.g., accuracy vs. Mean Absolute Error), and personality models, complicating direct comparisons between algorithms.
The Computational Personality Recognition Shared Task was established to address these gaps. By providing a common framework and "gold standard" data—where text is paired with validated psychological test results—the workshop enabled a rigorous evaluation of diverse strategies for predicting the "Big Five" personality traits: Extraversion, Neuroticism, Agreeableness, Conscientiousness, and Openness.
Methodology
Participants were invited to build systems for automatic personality classification using two distinct datasets:
- Essays: Approximately 2,400 stream-of-consciousness texts written by students, representing formal yet unstructured writing.
- myPersonality: Status updates from 250 Facebook users, including social network metrics, representing informal digital interaction.
The systems generally followed two philosophical approaches:
- Bottom-up (Data-Driven): These models identify patterns in raw n-grams without prior psychological assumptions, often using probabilistic models like Naïve Bayes.
- Top-down (Theory-Driven): These models leverage external psycholinguistic resources, such as LIWC (Linguistic Inquiry and Word Count) or the MRC database, to identify word categories known to correlate with personality traits.
Results and Discussion
The shared task revealed that feature selection and theoretical grounding are critical for performance. The most successful systems didn't merely process raw data but employed sophisticated ranking algorithms to prune the feature space.
As shown in the table above, Team Ma (Markovic et al.) achieved the highest F-measure of .904 on the myPersonality dataset using a combination of SVMs and Boosting. This success underscores the importance of their feature selection process. Generally, "top-down" approaches utilizing established psychological lexicons outperformed purely "bottom-up" methods, suggesting that dictionaries capturing sentiment or social orientation provide a stronger signal than simple word frequencies.
Comparison with Literature
To contextualize these findings, it is useful to compare them with previous efforts in the field. Prior research has varied significantly in terms of algorithms, evaluation metrics, and dataset sizes.
While many teams in the Shared Task outperformed majority-class baselines, the margins remain slim. Personality recognition is inherently difficult due to feature sparsity; unlike sentiment analysis, where specific words provide clear signals, personality is reflected in subtle patterns across large volumes of text.
Conclusion and Future Work
The Shared Task demonstrated that while personality signals can generalize across domains (e.g., training on Essays and testing on Facebook data), the predictive features remain relatively "weak." Future work should focus on exploring new lexical resources and deepening the theoretical links between digital behavior and psychology. The datasets released for this task will remain available as a permanent benchmark to foster continued innovation in the field.
Summary updated