Where Personality Classifiers Break Across Borders

Language models of personality read a person's posts and predict traits such as extraversion or the Myers-Briggs type. They are trained almost entirely on North American and British Reddit and Facebook users. When these classifiers are applied to English speakers in India, Nigeria, or the Philippines, nobody knows how much accuracy they lose. Nobody knows why either. A loss could come from three quite different sources. The first is that people who publicly post their test results in one country differ in trait mix from those in another, so the labels shift. The second is that the classifier has learned "writes like an American" as a shortcut for some trait, so it latches onto dialect. The third is that traits are genuinely expressed differently in language across cultures. These three causes call for different remedies: recalibration, removing the shortcut, or collecting local data.

This note proposes to measure all three causes separately for text-based personality inference. The study uses the location metadata already present in the PANDORA Reddit dataset. It adds a counterfactual test in which region-marking words in a user's text are swapped for equivalents from another region, and a same-person control that compares a user's posts in local versus global forums. The deliverable is a per-trait account of how much cross-region error each cheap fix removes and how much remains. It also includes a ranking of fixes that practitioners can use before deploying these models outside their home region.

Core Claim

Cross-region failure of text personality classifiers is not a single phenomenon, and the obvious remedies target different parts of it. We hypothesize that for most traits, the cross-region drop is dominated by two cheap-to-fix causes: shifted label base rates, and reliance on region-marking vocabulary. A smaller, trait-specific residual, which may concentrate in Extraversion and Agreeableness, requires local data. This is testable because each cause has its own correction, and the corrections can be applied in sequence and in reverse order.

This is new relative to the closest work in three ways. Modeling Personality vs. Modeling Personalidad ran leave-one-country-out tests across five countries. Its data was phone sensor logs from 166 people, not language, and it reported only aggregate accuracy changes, without separating label shift from feature shift. PANDORA Talks recorded location but audited bias only along gender. Detecting Cross-Geographic Biases in Toxicity Modeling supplies the counterfactual perturbation method and the evidence that Western-trained text models misread regional identity terms. It studied toxicity, not psychometrics.

Research Program

Region splits. We start from PANDORA: 10,288 Reddit authors, 17 million comments, with self-reported Big Five (normalized 0–100) and MBTI labels plus location. We group locations into regions coarse enough to have usable sample sizes. The likely groups are US, Canada, UK and Ireland, other Europe, South and Southeast Asia, and Oceania. The final grouping will be set by the counts we observe, and those counts will be reported.

For a secondary check on the MBTI dimensions, we infer region in MBTI9k from participation in country-specific subreddits. MBTI9k is the dataset of 13,631 users from Reddit: A Gold Mine for Personality Prediction. We validate this inference against PANDORA users for whom both signals exist.

Models. We train the models the literature shows are hard to beat. The first is L2-regularized logistic regression on word n-grams over long user histories, which PANDORA found beats BERT limited to short histories. The second is a transformer fine-tuned on the same long histories using chunked averaging.

The three corrections. Each targets one cause of cross-region error.

Prior correction addresses shifted base rates. It re-estimates the target region's label distribution from a small labeled sample, or with an unsupervised expectation–maximization estimate, and adjusts output probabilities by Bayes' rule.

Marker removal addresses dialect shortcuts. First we identify region-marking vocabulary: place names, currencies, local institutions, spelling variants, and regional slang. We find these words using the Dirichlet-prior log-odds method from the toxicity study, applied to region rather than toxicity. We then either mask these tokens or swap each one for a mapped equivalent from the training region, such as "lakh" to "hundred thousand" or "colour" to "color."

Local adaptation addresses genuine expression differences. It fine-tunes on a small labeled sample from the target region.

Decomposing the error. Write $\Delta$ for the gap between in-region and out-of-region performance. We report how much of $\Delta$ each correction closes when applied first, and when applied after the others. With three corrections this gives six orderings. We summarize them as Shapley shares, meaning each correction's contribution averaged over orderings, so the attribution does not depend on one arbitrary order.

Shortcut reliance score. We measure this directly as the mean absolute change in predicted trait probability when only region markers are swapped. A classifier that depends on genuine psychological signal should barely move.

Same-person control. This design borrows the platform-shift observation from VIKI, which found that 78.2% of users shifted at least one point on a Big Five trait between LinkedIn and X. For users who post both in their country's subreddit and in global subreddits, we predict traits separately from each set of posts. The label is identical across the two sets, so any systematic prediction difference is a context and dialect effect, not a trait difference. This separates "the region's people differ" from "the region's register differs" inside a single person.

Baselines

Do nothing. The naive reference is a North-America-trained n-gram logistic regression applied unchanged to each other region. Its in-region scores anchor $\Delta$. For scale, the MBTI9k authors report per-dimension macro F1 from 67% (Thinking/Feeling) to 82% (Sensing/Intuition), and 41% exact 16-type accuracy, in pooled evaluation (Reddit: A Gold Mine for Personality Prediction). For PANDORA itself, no reported personality-prediction result was found on Lacuna, so we will establish in-region PANDORA numbers ourselves before measuring any drop.

Strongest existing cross-region method. This is country-agnostic pooling with leave-one-region-out evaluation, which is exactly Method 1 of Modeling Personality vs. Modeling Personalidad. That paper reported 63% (Agreeableness) to 71% (Extraversion) binary accuracy on unseen countries, and a 3–7% gain from adding country-specific data for Extraversion, Agreeableness, and Conscientiousness. Those figures come from sensors, not text, so they motivate the per-trait hypothesis but are not a directly comparable target.

We run this baseline, and every other method, on identical user splits. All methods get the same hyperparameter search budget, tuned on validation users from the training regions only. Median-split binary labels are defined on the training regions. Local adaptation for every method uses the same small labeled target sample.

Hierarchical meta-learning. A stronger adaptive competitor is meta-learning across regions in the style of Adaptive-MAML-B from Task-Adaptive Meta-Learning Framework for Advancing Spatial Generalizability. That work raised hard-county R² from 37.96% to 64.39% on crop yield. We include it with regions as tasks and the same adaptation sample.

Zero-shot LLM. We prompt an LLM for trait ratings, averaged over ten runs as VIKI did. VIKI reports no accuracy against questionnaire labels, so no reported result was found on Lacuna for LLM trait-inference accuracy.

Key ablation. The ablation that isolates our central ingredient is marker removal alone versus prior correction alone.

What counts as success. Beating the baselines here means one of two outcomes. The combined cheap corrections could close a majority of $\Delta$ with bootstrap confidence intervals excluding zero, matching local adaptation or meta-learning without their labeled-data cost. Alternatively, if they do not, the decomposition shows where and for which traits the residual lives.

Why not just…?

Why not just run leave-one-region-out and report the drop? That measurement is our first experiment, but on its own it cannot tell a practitioner what to do. A 5-point drop caused by base-rate shift is fixed by recalibration using a few dozen labels. The same drop caused by dialect shortcuts needs feature changes. The sensor study reported only aggregate gains, which is why its "culture-robust" traits cannot yet be compared with the text findings of Bilingual Word Embeddings for Cross-Lingual Personality Recognition. That paper found zero-shot English-to-Chinese transfer strongest on Extraversion, Conscientiousness, and Neuroticism, which is a different set of traits.

Why not just mask place names and slang and retrain? This is a real threat. If masking alone closes most of $\Delta$, that is the headline finding, and the method reduces to a vocabulary audit. The Shapley decomposition and the swap-flip score are designed to reveal exactly this outcome. The toxicity study constrains our optimism: deleting biased examples and fine-tuning on balanced sets failed there, suggesting geographic associations can live in representations beyond explicit tokens. That is why we test both masking and swapping, and both the n-gram and transformer models.

Why not just balance age and gender, or condition on demographics? In the sensor study, demographic balancing gave up to 17% gains, yet the need for country-specific data remained. Domain Transfer for Empathy, Distress, and Personality Prediction conditioned on a 135-dimensional demographic vector and still produced a negative Pearson correlation for Extraversion (−0.1966). We include demographic reweighting on PANDORA's age and gender fields as an additional correction in the decomposition, so its share is measured rather than assumed.

Why not just fine-tune on local data? It will likely work best per region, and we treat it as the upper reference. The question is how much of its gain cheaper, label-light fixes recover. It also requires labeled local users, who are scarce precisely in the regions where transfer matters.

Proposed Experiments

The first experiment establishes in-region and out-of-region performance for both model families. For MBTI dimensions we report macro F1. For Big Five we report Pearson correlation and binary AUROC, along with expected calibration error. Confidence intervals come from user-level bootstrap resampling.

The second experiment is the ordered decomposition, reported per trait and per region.

Decisive checks. These determine how the results should be read. If the swap-flip score is near zero while $\Delta$ is large, dialect shortcuts are ruled out as the main cause. If prior correction alone closes most of $\Delta$, the problem is label selection, echoing the self-disclosure bias PANDORA already documents. The same-person control is the sharpest test. A systematic within-user prediction shift between local and global subreddits means the classifier reads register rather than traits. A near-zero shift, combined with a residual that survives all cheap corrections, is evidence of genuine cross-cultural differences in how traits appear in language.

Traits and modality. We compare the per-trait residuals with the sensor study's split: Neuroticism and Openness robust, the other three traits needing local data. This tells us whether cultural robustness belongs to the trait or to the observation modality.

Main risk and fallback. The main risk is small non-US samples. If regional groups fall below a few hundred labeled users, we pool them further. We would then report the analysis as a within-anglosphere versus outside-anglosphere contrast and state the resulting loss of resolution explicitly.