Workshop on Computational Personality Recognition: Shared Task

Fabio Celli, Fabio Pianesi, David Stillwell, Michał KosińskiAAAI 2021 · DOI

Abstract

In the Workshop on Computational Personality Recognition (Shared Task), we released two datasets, varying in size and genre, annotated with gold standard personality labels. This allowed participants to evaluate features and learning techniques, and even to compare the performances of their systems for personality recognition on a common benchmark. We had 8 participants to the task. In this paper we discuss the results and compare them to previous literature.

Introduction

Predicting personality from digital footprints—such as social media posts or stream-of-consciousness essays—is a rapidly expanding field with applications in recommendation systems and mental health monitoring. However, the lack of standardized benchmarks has historically hindered progress. Researchers often utilized disparate datasets, evaluation metrics (e.g., accuracy vs. Mean Absolute Error), and personality models, complicating direct comparisons between algorithms.

The Computational Personality Recognition Shared Task was established to address these gaps. By providing a common framework and "gold standard" data—where text is paired with validated psychological test results—the workshop enabled a rigorous evaluation of diverse strategies for predicting the "Big Five" personality traits: Extraversion, Neuroticism, Agreeableness, Conscientiousness, and Openness.

Methodology

Participants were invited to build systems for automatic personality classification using two distinct datasets:

  1. Essays: Approximately 2,400 stream-of-consciousness texts written by students, representing formal yet unstructured writing.
  2. myPersonality: Status updates from 250 Facebook users, including social network metrics, representing informal digital interaction.

The systems generally followed two philosophical approaches:

  • Bottom-up (Data-Driven): These models identify patterns in raw n-grams without prior psychological assumptions, often using probabilistic models like Naïve Bayes.
  • Top-down (Theory-Driven): These models leverage external psycholinguistic resources, such as LIWC (Linguistic Inquiry and Word Count) or the MRC database, to identify word categories known to correlate with personality traits.

Results and Discussion

The shared task revealed that feature selection and theoretical grounding are critical for performance. The most successful systems didn't merely process raw data but employed sophisticated ranking algorithms to prune the feature space.

Table 2: Results of the Shared Task participants across different datasets and algorithms.

As shown in the table above, Team Ma (Markovic et al.) achieved the highest F-measure of .904 on the myPersonality dataset using a combination of SVMs and Boosting. This success underscores the importance of their feature selection process. Generally, "top-down" approaches utilizing established psychological lexicons outperformed purely "bottom-up" methods, suggesting that dictionaries capturing sentiment or social orientation provide a stronger signal than simple word frequencies.

Comparison with Literature

To contextualize these findings, it is useful to compare them with previous efforts in the field. Prior research has varied significantly in terms of algorithms, evaluation metrics, and dataset sizes.

Table 1: Summary of previous literature in computational personality recognition.

While many teams in the Shared Task outperformed majority-class baselines, the margins remain slim. Personality recognition is inherently difficult due to feature sparsity; unlike sentiment analysis, where specific words provide clear signals, personality is reflected in subtle patterns across large volumes of text.

Conclusion and Future Work

The Shared Task demonstrated that while personality signals can generalize across domains (e.g., training on Essays and testing on Facebook data), the predictive features remain relatively "weak." Future work should focus on exploring new lexical resources and deepening the theoretical links between digital behavior and psychology. The datasets released for this task will remain available as a permanent benchmark to foster continued innovation in the field.

Summary updated

Abstract

In the Workshop on Computational Personality Recognition (Shared Task), we released two datasets, varying in size and genre, annotated with gold standard personality labels. This allowed participants to evaluate features and learning techniques, and even to compare the performances of their systems for personality recognition on a common benchmark. We had 8 participants to the task. In this paper we discuss the results and compare them to previous literature.
BibTeX
@inproceedings{celli2021workshop,
  author = {Fabio Celli and Fabio Pianesi and David Stillwell and Michał Kosiński},
  title = {{Workshop on Computational Personality Recognition: Shared Task}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v7i2.14467},
}
Figure