Understanding the Political Ideology of Legislators from Social Media Images

Nan Xi, Di Ma, Marcus Liou, Zachary C. Steinert-Threlkeld, Jason Anastasopoulos, Jungseock Joo

TL;DR

By analyzing Facebook photos posted by members of the 114th U.S. Congress with deep convolutional neural networks, this study demonstrates that political ideology can be accurately estimated from imagery alone. The model achieves an 82.35% classification accuracy when aggregating photos per politician, and its predictions correlate strongly ($r = 0.67$) with official DW-NOMINATE legislative voting scores.

What Problem Does This Paper Address?

In political science, measuring the ideology of politicians has historically relied on roll-call voting records (e.g., the NOMINATE algorithm) or textual transcripts of speeches and party platforms. However, modern political rhetoric is visually saturated: virtually every legislator actively uses platforms like Facebook and Instagram to shape public perceptions.

Images elicit rapid, emotionally driven "System 1" cognitive responses, yet their ideological content has been difficult to systematically quantify at scale. The authors investigate whether machine learning can reliably infer a politician’s ideology solely from the photos they share, how computational models compare to human perception, and what specific visual elements characterize liberal versus conservative visual messaging.

Methods

The authors designed a multi-step computer vision pipeline:

  • Ideology & Party Classification: A ResNet-34 architecture (pretrained on ImageNet) was fine-tuned to classify images as Democrat or Republican. To avoid data leakage and prevent the model from memorizing specific faces or venues, evaluation was performed using 10-fold cross-validation split by politician.
  • Demographic Control: Because party affiliation is heavily confounded by demographic disparities (e.g., Congress has far more female and minority Democrats than Republicans), the classification was restricted to white male legislators (68 per party, 150 randomly sampled photos each).
  • Unsupervised Face Recognition & Facial Attribute Analysis: Faces were detected using dlib and clustered using Chinese-Whispers clustering on 128-dimensional embeddings to automatically separate the politician from other people ("associates"). Separate ResNet-34 models trained on public face benchmarks classified facial expressions (ExpW), binary attributes (CelebA), and perceived race/gender (UTKFace).
  • Semantic & Visual Saliency Analysis: Visual elements were analyzed using the Google Vision API (evaluated via $\chi^2$ association tests) and Grad-CAM saliency maps.
  • Human Benchmark: 272 randomly sampled images were evaluated by crowd-workers on Amazon Mechanical Turk (50 workers per image) to establish a human baseline.

Datasets

  • Congressional Facebook Dataset (Introduced): Collected in March 2016, covering 296,461 images from 319 members of the 114th U.S. Congress (2015–2017). After excluding infographics and accounts with fewer than 150 photos, the primary experimental dataset consisted of 20,400 photos across 136 white male legislators.
  • Facial Benchmarks: Pretrained models utilized ExpW (106,962 facial expression images), CelebA (202,599 attribute-labeled images), and UTKFace (24,108 demographic-labeled faces).

Key Insights and Ideas

  • Visuals Map Directly to Voting Behavior: Image-predicted ideology is not just a binary party detector; it mirrors continuous ideological variation. Aggregated visual predictions correlate significantly with actual roll-call voting records ($r = 0.6736$ overall; $r \approx 0.23$ to $0.26$ within each individual party).
  • Status Quo vs. Social Change: Visual themes closely align with political psychology theories of ideology. Conservative images emphasize status quo maintenance, authority, and patriotism (military officers, uniforms, banquets, flags, courts). Liberal imagery emphasizes social change, environmental protection (nature reserves, coastal scenes), and underrepresented or low-wage groups.
  • The "Happiness" Gap: Republicans display significantly more happiness (smiling) than Democrats (71.4% vs. 65.4% for the politicians; 66.5% vs. 62.7% for accompanying associates). This directly contradicts prior work analyzing official directory portraits and instead supports survey research showing conservatives self-report higher life satisfaction.
  • Humans vs. Machines: Deep learning models slightly edge out human annotators on single-image classification (63.6% vs. 61.0%). While humans excel at contextual real-world deduction (e.g., recognizing that a red shirt belongs to Habitat for Humanity volunteers rather than the GOP), they lean heavily on race and gender stereotypes. The CNN, in contrast, picks up on subtle recurring institutional backdrops (such as the Capitol dome).

Key Results

  • Classification Accuracy: 59.28% ($\text{SD} = 2.00$) on individual photographs; jumping to 82.35% when averaging scores across a politician’s photo timeline (up to 150 images).
  • Ideological Alignment: A Pearson correlation of $r = 0.6736$ ($p = 7.79 \times 10^{-25}$) between model predictions and official DW-NOMINATE scores.
  • Associate Demographics: People appearing alongside Democratic politicians were significantly more likely to be non-white (28.8% vs. 17.1%, $p < 0.0001$) and female (37.0% vs. 35.5%, $p = 0.032$) compared to those pictured with Republicans.

Limitations and Future Work

  • Demographic Homogeneity: To control for race and gender confounders between the parties, the core modeling was strictly limited to white male legislators, leaving open how women and minority politicians navigate visual ideological signaling.
  • Infographic Filtering: The study discarded text-heavy infographics, which are increasingly central to modern digital campaign strategies.
  • Causality and Persuasion: The findings characterize correlations in posted imagery rather than the causal impact these visual cues have on constituent persuasion, polarization, voter turnout, or fundraising.

Summary updated

Abstract

In this paper, we seek to understand how politicians use images to express ideological rhetoric through Facebook images posted by members of the U.S. House and Senate. In the era of social media, politics has become saturated with imagery, a potent and emotionally salient form of political rhetoric which has been used by politicians and political organizations to influence public sentiment and voting behavior for well over a century. To date, however, little is known about how images are used as political rhetoric. Using deep learning techniques to automatically predict Republican or Democratic party affiliation solely from the Facebook photographs of the members of the 114th U.S. Congress, we demonstrate that predicted class probabilities from our model function as an accurate proxy of the political ideology of images along a left–right (liberal–conservative) dimension. After controlling for the gender and race of politicians, our method achieves an accuracy of 59.28% from single photographs and 82.35% when aggregating scores from multiple photographs (up to 150) of the same person. To better understand image content distinguishing liberal from conservative images, we also perform in-depth content analyses of the photographs. Our findings suggest that conservatives tend to use more images supporting status quo political institutions and hierarchy maintenance, featuring individuals from dominant social groups, and displaying greater happiness than liberals.
BibTeX
@inproceedings{xi2020understanding,
  author = {Nan Xi and Di Ma and Marcus Liou and Zachary C. Steinert-Threlkeld and Jason Anastasopoulos and Jungseock Joo},
  title = {{Understanding the Political Ideology of Legislators from Social Media Images}},
  booktitle = {ICWSM},
  year = {2020},
  url = {https://openreview.net/forum?id=WEXxN6XJuj6j},
}

Loading related content...

Figure