Abstract
Text-based Personality Computing (TPC) has seen a surge in interest as researchers leverage Natural Language Processing (NLP) to infer psychological traits from digital footprints. However, this growth has often come at the expense of psychological rigor. This paper provides a systematic review of 60 empirical studies from the ACL Anthology, identifying 15 critical challenges across six core themes: taxonomies, measurement quality, datasets, performance evaluation, modelling choices, and ethics. By bridging the gap between NLP methodology and psychometric foundations, this work provides a roadmap for more valid, reliable, and ethical personality models.
Introduction
In recent years, the ability to predict human personality from text—ranging from social media posts to interview transcripts—has become a prominent subfield of NLP. While deep learning has significantly improved predictive performance, the field often treats personality computing as a standard classification or regression task, ignoring the complex reality of psychological measurement. Most TPC research operates under the assumption that questionnaire scores are an infallible "gold standard." In reality, these scores are noisy proxies for latent psychological constructs. This disconnect between computational methods and social science foundations risks producing models that lack scientific validity or, worse, pose ethical hazards.
Methodology
To map the current landscape of TPC, a systematic literature search was conducted within the ACL Anthology. The review analyzed 60 empirical papers through an interdisciplinary lens, combining NLP techniques with psychometrics—the science of psychological measurement. The findings are categorized into 15 specific challenges (C1–C15) grouped into six thematic areas: taxonomies, measurement quality, dataset availability, evaluation metrics, modelling strategies, and ethical considerations.
Taxonomies: Big-5 vs. MBTI
A fundamental choice in TPC is the selection of a personality framework. The review found that 45 out of 60 papers utilized the Big-5 (OCEAN) model, while 14 utilized the Myers-Briggs Type Indicator (MBTI).
The authors strongly advocate for the Big-5 model over MBTI. While MBTI is popular in commercial and non-academic settings, it is purely theory-driven and lacks empirical support. In contrast, the Big-5 is rooted in the "lexical hypothesis" and treats personality as a continuous spectrum. TPC research that relies on MBTI's binary "boxes" often loses significant nuance and scientific credibility.
The Challenge of Measurement Quality
A critical insight from this review is that the "gold standard" labels used in TPC are often "bronze" at best. Questionnaire scores contain both random error (e.g., participant fatigue or misinterpretation) and systematic error (e.g., poorly phrased items). When NLP models are trained on this data without accounting for measurement error, they inevitably propagate these inaccuracies.
Furthermore, the field suffers from a lack of item-level data. Most shared datasets provide only aggregated scores (e.g., a single score for "Extraversion"). Without raw item-level responses, researchers cannot assess the internal consistency or reliability of the ground truth labels they are using to train their models.
Dataset Landscape and Accessibility
The review identified 41 unique datasets used across the 60 papers. However, a significant barrier to progress is data accessibility: only 18 of these datasets are potentially accessible to the broader research community. Notable shareable benchmarks include:
- PANDORA: A large-scale dataset based on Reddit.
- WASSA 2022: Focused on empathy and personality in reactions to news.
- Personal-ITY: YouTube comments labeled with MBTI.
- Twitter (Plank & Hovy, 2015): A standard social media benchmark.
- Essays (Pennebaker & King, 1999): A classic dataset of student writing.
Modeling and Evaluation
Most TPC studies model personality traits (e.g., Openness, Neuroticism) in isolation. However, psychological research shows these traits are naturally correlated. Only 5 out of 60 studies attempted joint personality modelling (e.g., multitask learning). Leveraging these relationships can lead to more robust and psychologically grounded models.
Evaluation metrics also require refinement. While many studies treat TPC as a regression task, they often report Pearson’s correlation. The authors point out that correlation only measures rank and can be misleading regarding actual predictive accuracy; they recommend the adoption of $R^2$ as a more rigorous metric.
Ethics and Fairness
Despite the sensitive nature of personality data, ethical oversight in the field remains sparse. Only 7 out of 60 papers provided any reflection on ethics, and none addressed algorithmic fairness or demographic bias. There is a pressing need to investigate whether TPC models perform equitably across different ages, genders, and ethnicities, and to develop frameworks for "data donation" that allow users to share digital traces safely.
Conclusion and Future Directions
To move TPC forward, the field must transition from "predicting tweets" to meaningful applications in clinical psychology and education. This requires a shift toward:
- Item-Level Modeling: Releasing raw questionnaire data for better psychometric analysis.
- Fairness Research: Ensuring models do not reinforce demographic biases.
- Privacy-Preserving Frameworks: Developing ethical pipelines for data acquisition.
By addressing these 15 challenges, TPC can evolve from a black-box engineering task into a scientifically rigorous tool for understanding human behavior.
Summary updated