Large-Scale, Longitudinal Study of Large Language Models During the 2024 US Election Season

Sarah H. Cen, Andrew Ilyas, Hedi Driss, Charlotte Park, Aspen K. Hopkins, Chara Podimata, Aleksander MadryarXiv 2025 · arXiv · OpenReview

Abstract

The 2024 US presidential election is the first major contest to occur in the US since the popularization of large language models (LLMs). Building on lessons from earlier shifts in media (most notably social media's well studied role in targeted messaging and political polarization) this moment raises urgent questions about how LLMs may shape the information ecosystem and influence political discourse. While platforms have announced some election safeguards, how well they work in practice remains unclear. Against this backdrop, we conduct a large-scale, longitudinal study of 12 models, queried using a structured survey with over 12,000 questions on a near-daily cadence from July through November 2024. Our design systematically varies content and format, resulting in a rich dataset that enables analyses of the models' behavior over time (e.g., across model updates), sensitivity to steering, responsiveness to instructions, and election-related knowledge and "beliefs." In the latter half of our work, we perform four analyses of the dataset that (i) study the longitudinal variation of model behavior during election season, (ii) illustrate the sensitivity of election-related responses to demographic steering, (iii) interrogate the models' beliefs about candidates' attributes, and (iv) reveal the models' implicit predictions of the election outcome. To facilitate future evaluations of LLMs in electoral contexts, we detail our methodology, from question generation to the querying pipeline and third-party tooling. We also publicly release our dataset at https://huggingface.co/datasets/sarahcen/llm-election-data-2024

Introduction

The 2024 US election represents a pivotal moment in the intersection of artificial intelligence and democratic processes. As LLMs become mainstream information tools, they act as conversational authorities, potentially influencing political discourse in ways distinct from the social media-driven polarization of previous decades. This study addresses the impact of LLMs by tracking 12 major models over a four-month period, revealing how they evolve as an election approaches, how they respond to user "nudges" (demographic steering), and what their responses reveal about internal "beliefs" regarding candidates and outcomes.

Methodology and Data

To systematically evaluate model behavior, the researchers developed a querying pipeline that surveyed 12 models—including versions of GPT-4, Claude 3.5, and Gemini 1.0—on a near-daily cadence from July through November 2024. The core of this methodology is a structured taxonomy of over 12,000 queries.

Table 1: Taxonomy of queries used in the longitudinal study, categorized by type and frequency.

As shown in Table 1, queries were categorized into three primary types:

  • Endogenous: Questions about stable concepts (e.g., "How does the Electoral College work?") where responses should remain consistent.
  • Exogenous: Questions tied to current events (e.g., "Who is the current frontrunner?") where responses are expected to shift over time.
  • Baseline: Standard benchmarks (GSM8k, NaturalQA, MedQA) used to ensure that observed shifts were specific to election content rather than general model degradation.

The study also employed Demographic Steering, testing 22 prompt variations by prepending identities (e.g., "I am a Republican") to measure shifts in tone or content. For models that refused direct predictions, an Implicit Belief Extraction method was used, deducing model "views" through indirect questions about voter groups and exit polls.

Key Insights and Results

The longitudinal analysis revealed several critical phenomena:

The "Step" Change Phenomenon

Models do not merely drift in their responses; they often undergo abrupt "step" changes. Sharp, persistent shifts in behavior were identified on specific dates, such as September 3 and October 2. While some shifts aligned with announced updates (like GPT-4o), others were unannounced, suggesting "silent" deployments of new guardrails or fine-tuning during the election cycle.

Heterogeneity in Steerability

The sensitivity to user identity varies significantly across providers. Gemini 1.0 Pro was identified as the most "steerable," showing high sensitivity to demographic prompts. Conversely, GPT-4 demonstrated the most resistance to such nudges. Interestingly, for GPT-4o, political identity ("I am a Republican" or "I am a Democrat") elicited more distinct responses than any other demographic category, such as race or gender.

Candidate Associations and Implicit Beliefs

Models exhibited clear patterns in candidate perception. Donald Trump was frequently associated with terms like "divisive" and "corrupt," while Kamala Harris was linked to "compassionate" and "honorable." However, models were not always self-consistent; a model might implicitly predict a win for one candidate on education issues but for another on government efficiency, indicating that LLMs lack a singular, unified "worldview."

Discussion and Limitations

While this study provides a comprehensive look at LLM behavior, several limitations exist. The use of API access may differ from consumer chatbot experiences, which often include additional system prompts. Furthermore, some online-augmented models provided incomplete responses due to token limits. Most importantly, while the study measures what models say, it does not quantify the downstream effects on human voting behavior. Future research is needed to understand how these model biases and shifts actually translate into real-world political influence.

The complete dataset, LLM Election Data 2024, is publicly available to facilitate further evaluation of AI in electoral contexts.

Summary updated

Abstract

The 2024 US presidential election is the first major contest to occur in the US since the popularization of large language models (LLMs). Building on lessons from earlier shifts in media (most notably social media's well studied role in targeted messaging and political polarization) this moment raises urgent questions about how LLMs may shape the information ecosystem and influence political discourse. While platforms have announced some election safeguards, how well they work in practice remains unclear. Against this backdrop, we conduct a large-scale, longitudinal study of 12 models, queried using a structured survey with over 12,000 questions on a near-daily cadence from July through November 2024. Our design systematically varies content and format, resulting in a rich dataset that enables analyses of the models' behavior over time (e.g., across model updates), sensitivity to steering, responsiveness to instructions, and election-related knowledge and "beliefs." In the latter half of our work, we perform four analyses of the dataset that (i) study the longitudinal variation of model behavior during election season, (ii) illustrate the sensitivity of election-related responses to demographic steering, (iii) interrogate the models' beliefs about candidates' attributes, and (iv) reveal the models' implicit predictions of the election outcome. To facilitate future evaluations of LLMs in electoral contexts, we detail our methodology, from question generation to the querying pipeline and third-party tooling. We also publicly release our dataset at https://huggingface.co/datasets/sarahcen/llm-election-data-2024
BibTeX
@misc{cen2025large,
  author = {Sarah H. Cen and Andrew Ilyas and Hedi Driss and Charlotte Park and Aspen K. Hopkins and Chara Podimata and Aleksander Madry},
  title = {{Large-Scale, Longitudinal Study of Large Language Models During the 2024 US Election Season}},
  year = {2025},
  eprint = {2509.18446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18446},
}
Figure