Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms

Oleg Vasilyev

TL;DR

Researchers tested five popular LLM configurations over 30 days to see if their evaluations of U.S. presidential actions and hypothetical reforms are stable. They found that while models generally agree on the direction and ranking of policy impacts, "silent" updates to the underlying services can cause sudden, unannounced shifts in their reasoning and scores.

What Problem Does This Paper Address?

As LLMs become ubiquitous, people are increasingly using them to help navigate complex public affairs—from summarizing news to deliberating on policy. However, for an AI’s judgment to be useful, it must be robust. If a chatbot gives you a glowing review of a policy today but a scathing one tomorrow (or if Claude disagrees fundamentally with Gemini), the tool becomes unreliable for serious decision support.

Most existing research focuses on a model's "political leanings" via forced-choice surveys. This paper shifts the focus to evaluative judgments: asking models to score the long-term (50-year) consequences of specific, real-world actions. The authors tackle the "black box" nature of consumer interfaces, where users have no control over model versions, routing, or system prompts, to see if these tools provide consistent answers in a practical, everyday setting.

Methods

The study employed a rigorous, longitudinal setup:

  • The Subjects: Five chatbot configurations: Mistral (Think mode), Gemini Flash 3.6, Claude Sonnet 5, and Claude Opus 5 (at two different "effort" settings).
  • The Task: Every day for 30 days, each model was asked twice (in fresh, temporary chats) to score 40 real U.S. presidential actions (Obama through Trump’s second term) and 15 hypothetical reforms.
  • The Metrics: Models provided a score (-5 to +5 for actions), a confidence level (1-5), and a short explanatory comment.
  • Analysis: The authors measured "drift" over time and "noise" between same-day runs. They also treated the explanatory comments as a retrieval task, using lexical similarity (Jensen-Shannon divergence) and multiple embedding models (Jina, Harrier, Granite) to see if the models' reasoning remained consistent.
Figure 1: All 600 prompt runs: 5 chatbot configurations × 2 prompts × 30 days × 2 runs a day, shown as small boxes. The setup is explained in Section 2.

Datasets

The authors introduced a new public dataset:

  • 16,500 Judgments: A collection of scores, confidence levels, and comments for 40 presidential actions and 15 hypothetical reforms.
  • Action/Reform Metadata: Detailed descriptions of the items judged, ranging from the Affordable Care Act to hypothetical "mass deportation" campaigns or "universal tariff" regimes.
  • The dataset is available on Hugging Face for further community analysis.

Key Insights and Ideas

  • Surprising Consensus: Despite different architectures and training data, all models agreed on the "sign" (positive vs. negative) and the relative ranking of the four administrations' mean scores.
  • The "Silent Update" Risk: The most significant finding was the identification of three abrupt shifts in model behavior that occurred without any change to the user interface. For example, Claude Sonnet's reported "knowledge cutoff" flipped between January and August 2026 on specific days, accompanied by changes in comment length and scoring patterns.
  • Reasoning is Specific: The explanatory comments weren't just generic "AI-speak." They were highly specific to each policy. Using standard retrieval techniques, the authors could correctly identify which policy a comment belonged to with very high accuracy (MAP > 0.92).
  • Model Choice Matters More Than Time: A model’s responses 25 days apart were generally closer to each other than to a different model’s response on the same day. Switching from Claude to Gemini changes the "opinion" more than waiting a month.

Key Results

  • Administration Rankings: All models ranked the net long-term flourishing impact of administrations in the same order: Obama (highest), followed by Biden, Trump’s first term, and Trump’s second term (lowest).
  • Low Disagreement: Two different models asked about the same action on the same day gave opposite signs (one positive, one negative) in only about 4% of cases.
  • Confidence Trends: Models were most confident about older actions (Obama era) where more historical evidence exists, and least confident about the most recent or hypothetical ones.
Mean scores and confidence levels across administrations show a clear, consistent trend across all models.
Figure 4: Summary of Figure 3. Left: for each admin- istration, the bar spans the lowest and the highest of the five models’ mean scores, and the dark line is their average. Right: the same for the confidence, with the confidence in the hypothetical reforms (R) added on the right.

Limitations and Future Work

  • Time Horizon: 30 days is a blink of an eye compared to the 50-year impact horizon the models were asked to judge.
  • Robustness $\neq$ Correctness: The authors emphasize that just because models agree doesn't mean they are right. Models trained on similar data may simply share the same biases or errors.
  • Prompt Sensitivity: The study used a single prompt in English. Future work could explore how different phrasings or languages affect these judgments.
  • Selection Bias: The actions and reforms were selected with the help of Claude Opus, which might have influenced the "consequentiality" of the items included in the benchmark.

Summary updated

Abstract

Chatbots are increasingly consulted about public affairs, and their judgments of policies are useful only if they are robust. We measure this robustness in the setting of an ordinary user, through consumer chat interfaces. For 30 days, five chatbot configurations (Mistral, Gemini Flash, Claude Sonnet, and Claude Opus at two effort settings) were asked twice a day to score the long-term effect on the United States of 40 actions of the last four presidential administrations and of 15 hypothetical reforms, each with a confidence and a short comment. The judgments are robust: all configurations agree on the sign and the ranking of the administrations' mean scores, and two different configurations asked on the same day give opposite signs to the same action in about 4% of cases. Responses from different days differ more than same-day responses, but the difference grows slowly over the month, except where the service itself changed. We identify three such changes, which shifted the judgments while leaving little or no trace in the interface. Treating the comments as a retrieval task, we find them specific to each item and consistent across runs and chatbots, with lexical and embedding similarities agreeing. We release the dataset of 16,500 judgments.
BibTeX
@misc{vasilyev2026robustness,
  author = {Oleg Vasilyev},
  title = {{Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms}},
  year = {2026},
  eprint = {2610.39398},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39398},
}
Figure