Analyzing Biases in Human Perception of User Age and Gender from Text

Lucie Flekova, Jordan Carpenter, Salvatore Giorgi, Lyle H. Ungar, Daniel Preotiuc-Pietro

Introduction

In the development of personalized AI—such as recommender systems and conversational agents—understanding user demographics like age and gender is a primary objective. However, a significant gap often exists between a user's actual identity and how they are perceived by others based on their digital footprint. This study investigates the discrepancy between "ground truth" (self-reported data) and "perceived traits" (human-annotated data) using Twitter as a case study. The researchers explore human accuracy in judging strangers from short text snippets, the stereotypes that drive these judgments, and the potential for these human biases to enhance machine learning models.

Methodology

The study employed a multi-faceted approach to quantify human perception:

  1. Crowdsourcing: Over 1,000 Amazon Mechanical Turk annotators guessed the age and gender of Twitter users based on 20-tweet samples, providing confidence ratings for each guess.
  2. Statistical Modeling: A Bag-of-Words (BoW) classifier using Support Vector Machines (SVM) was established as a baseline for purely statistical prediction.
  3. Feature Analysis: The researchers used unigram analysis, topic modeling, and the NRC Emotion Lexicon to identify linguistic cues correlated with perceived versus actual traits.
  4. Hybrid Modeling: A combined model was tested to see if integrating human perceptions with statistical features could outperform standalone systems.

The datasets included 2,607 users with linked gender profiles and 826 users who self-reported their birth year. Tweets were anonymized to ensure judgments relied solely on language style rather than explicit metadata.

Key Results and Human Accuracy

The findings reveal that while humans are generally capable of identifying demographic traits from text, their accuracy is significantly improved through collective intelligence.

Accuracy improves as more human votes are aggregated, reaching over 85% for gender classification.

For gender, individual accuracy was 75.7%, which rose to 85.8% when using a majority vote of nine annotators. Age prediction showed a correlation of $r = 0.631$ between perceived and real age, with a Mean Absolute Error (MAE) of 6.05 years. Interestingly, a hybrid model combining human guesses with the SVM achieved 87.7% accuracy for gender, suggesting that human intuition and machine statistics capture complementary information. Humans are sensitive to nuance and social context, while machines are better at picking up on subtle, non-stereotypical statistical distributions of common words.

Systematic Biases and Stereotypes

A central insight of the study is that human errors are not random but systematic, driven by "exaggerated truths" or stereotypes. Humans use cues that are statistically linked to a group but over-apply them. For example, while both men and women talk about "love," humans are much more likely to label a post as "female" if that word appears, even when the data doesn't support such a strong distinction.

The Youthful Bias

One of the most prominent biases observed was the "Youthful Bias." Annotators consistently under-predicted the age of older Twitter users. This suggests a platform-specific stereotype where the act of tweeting itself is associated with youth.

The relationship between real and perceived age shows a clear trend of under-predicting the age of older individuals.

Gender Stereotyping

Humans often over-apply linguistic cues. Men using "warm" or "emotional" language were frequently mislabeled as female, highlighting how social expectations of gendered communication influence perception. The study found that humans are more likely to mislabel men as women than vice versa in these contexts. Conversely, annotators felt most confident when they saw "explicit" cues like mentions of family or beauty products, though high confidence did not always correlate with higher accuracy.

Conclusion

This research demonstrates that human perception of demographics from text is grounded in directionally correct but exaggerated stereotypes. While these biases lead to systematic errors—particularly the under-estimation of age for older users—they also contain valuable social nuance that can be leveraged to improve AI. By understanding the gap between identity and perception, developers can build more sophisticated models that better reflect how users are viewed in social contexts. Future work may apply this methodology to other traits like education level, race, or political orientation to see if similar patterns emerge.

Summary updated

BibTeX
@inproceedings{flekova2016analyzing,
  author = {Lucie Flekova and Jordan Carpenter and Salvatore Giorgi and Lyle H. Ungar and Daniel Preotiuc-Pietro},
  title = {{Analyzing Biases in Human Perception of User Age and Gender from Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/p16-1080},
}
Figure