Facial recognition technology can expose political orientation from naturalistic facial images

Michał KosińskiScientific 2021 · DOI

TL;DR

Facial recognition algorithms can predict an individual's political orientation with 72% accuracy from naturalistic facial images, significantly outperforming human judgment and even personality questionnaires. This capability, achieved across diverse datasets and countries, raises critical concerns about privacy and civil liberties due to the pervasive nature of facial recognition technology.

What Problem Does This Paper Address?

The paper addresses the growing concern that ubiquitous facial recognition technology, fueled by vast databases of images from social media and public records, poses a significant threat to privacy and civil liberties. Beyond simple identification, these algorithms can infer personal attributes. While humans can make some face-based judgments about personality or political leanings, their accuracy is limited. The authors investigate whether advanced facial recognition algorithms can achieve higher accuracy in predicting political orientation from naturalistic facial images, and what the implications of such capabilities are.

Methods

The researchers employed a facial recognition algorithm, specifically VGGFace2, to extract 2,048-dimensional "face descriptors" from facial images. These descriptors represent the core features of a face. To predict political orientation (liberal vs. conservative), these descriptors were compared against average face descriptors of known liberals and conservatives. A cross-validated logistic regression model was used to classify political orientation. Alternative methods, including a deep neural network classifier and a simple ratio of cosine similarities to liberal and conservative faces, were also tested and yielded similar results. The study also controlled for demographic factors like age, gender, and ethnicity by recomputing accuracies on subsets of data matched for these attributes.

Datasets

The study utilized a large dataset comprising 1,085,795 participants from three countries: the U.S., Canada, and the UK. The facial images were sourced from two primary environments:

  • Dating Websites: Images from a popular dating website, collected in 2017.
  • Facebook: Public profile images from U.S. Facebook users, collected between 2007 and 2012.

Participants provided self-reported political orientation, age, and gender. The datasets included a diverse ethnic representation, with over 347,000 non-white participants. The paper does not introduce new benchmark datasets but rather leverages existing, large-scale, naturalistic image collections.

Key Insights and Ideas

  • Faces as Political Fingerprints: The core insight is that political orientation leaves discernible, albeit subtle, traces on facial appearance that can be detected by sophisticated algorithms. This suggests a deeper, perhaps unconscious, link between our internal political beliefs and our outward presentation than previously understood.
  • Algorithmic Superiority: Facial recognition algorithms significantly outperform human judgment in predicting political orientation from faces. This highlights the power of machine learning to identify complex patterns invisible to the human eye.
  • Beyond Demographics: While demographic factors (age, gender, ethnicity) contribute to predictability, the accuracy remains high even when these are controlled for, indicating that other facial cues are crucial.
  • Privacy Implications: The study underscores a significant privacy threat. If political orientation can be reliably inferred from a face, then facial recognition technology can be used to profile individuals based on their political beliefs without their consent, potentially leading to discrimination or manipulation.

Key Results

  • High Classification Accuracy: The facial recognition algorithm achieved an average accuracy of 72% in classifying political orientation across liberal-conservative face pairs. This is substantially higher than chance (50%) and human accuracy (55%).
  • Cross-Sample and Cross-Country Consistency: Accuracy remained remarkably consistent across different countries (U.S., Canada, UK), environments (dating websites, Facebook), and when models trained on one dataset were applied to another (out-of-sample accuracy of 68%).
  • Robustness to Demographics: Even when controlling for age, gender, and ethnicity, the accuracy remained high, averaging around 68-69%. This suggests that cues beyond basic demographics are at play.
  • Superior to Personality Prediction: The facial recognition algorithm's accuracy (73% on the Facebook sample) surpassed that of a 100-item personality questionnaire (66%) in predicting political orientation.

Limitations and Future Work

  • Definition of Political Orientation: The study used a dichotomous classification (liberal vs. conservative), which is a simplification of complex ideological spectrums. More nuanced political classifications might yield different results.
  • Naturalistic Images: While naturalistic images are valuable for real-world relevance, they also introduce variability (lighting, expression, pose) that could be further exploited or mitigated.
  • Potential for Bias: The study acknowledges that demographic traits are correlated with political orientation. While they controlled for these, the underlying reasons for these correlations and their impact on algorithmic bias warrant further investigation.
  • Future Directions: The authors suggest that higher accuracy could be achieved by using multiple images per person, higher resolution images, custom neural networks, or incorporating non-facial cues like hairstyle and clothing. They also emphasize the ongoing "arms race" between human adaptation and algorithmic detection, suggesting that the ability to extract relevant information from faces will likely continue to improve.

Summary updated

Abstract

Ubiquitous facial recognition technology can expose individuals' political orientation, as faces of liberals and conservatives consistently differ. A facial recognition algorithm was applied to naturalistic images of 1,085,795 individuals to predict their political orientation by comparing their similarity to faces of liberal and conservative others. Political orientation was correctly classified in 72% of liberal-conservative face pairs, remarkably better than chance (50%), human accuracy (55%), or one afforded by a 100-item personality questionnaire (66%). Accuracy was similar across countries (the U.S., Canada, and the UK), environments (Facebook and dating websites), and when comparing faces across samples. Accuracy remained high (69%) even when controlling for age, gender, and ethnicity. Given the widespread use of facial recognition, our findings have critical implications for the protection of privacy and civil liberties.
BibTeX
@inproceedings{kosinski2021facial,
  author = {Michał Kosiński},
  title = {{Facial recognition technology can expose political orientation from naturalistic facial images}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-020-79310-1},
}
Figure