This document is a Pre-Analysis Plan (PAP) for an upcoming study. As such, it outlines the experimental design and intended hypotheses rather than reporting final experimental results.
TL;DR
Researchers are launching a large-scale randomized controlled trial to determine if AI chatbots (BallotBot) can improve political literacy and voter turnout compared to traditional static voter guides. By leveraging GPT-4o and Retrieval-Augmented Generation (RAG), the study tests whether interactive, personalized information retrieval lowers the "cognitive tax" of participating in direct democracy.
What Problem Does This Paper Address?
Direct democracy—where citizens vote directly on policy via ballot initiatives—places a heavy informational burden on the electorate. In states like California, official voter guides are often dense, legalistic, and difficult to navigate, leading to "rational ignorance" or voter fatigue.
The authors investigate whether new information technologies can bridge this gap. Specifically, they ask: Can an interactive chatbot make political information more accessible, reduce the perceived "cost" of becoming informed, and ultimately lead to more confident and active voters?
Methods
The study employs a two-wave online survey experiment with approximately 2,400 California residents recruited via Prolific.
- Randomization: Participants are randomly assigned to one of three real California ballot initiatives (Proposition 32, 33, or 36). Within those groups, they are split into two treatment arms:
- T1 (Control): Access to a digital, HTML version of the official California voter guide.
- T2 (Treatment): Access to BallotBot, an interactive chatbot powered by GPT-4o and an Azure-based information retrieval tool (RAG) trained exclusively on official ballot data.
- The Intervention: Participants use their assigned tool to answer five compulsory questions of varying difficulty (very easy to very hard).
- Economic Incentives: To ensure effort, participants earn $0.20 for every correct answer.
- Measuring Subjective Cost: The researchers use the Becker-DeGroot-Marschak (BDM) method, asking participants to state a "bid" (compensation) they would require to answer an additional difficult question. This serves as a proxy for the perceived effort or "cost" of acquiring knowledge.
- Wave 2: A follow-up survey weeks later measures retained knowledge and self-reported turnout behavior.
Datasets
- Primary Data: The study will generate a new dataset of survey responses, interaction logs from the chatbot, and voting intentions from ~2,400 participants.
- Source Material: BallotBot is grounded in official California statewide ballot measures.
- Evaluation Benchmarks: The researchers generated a pool of 12 questions per proposition, categorized by four difficulty levels, to benchmark political knowledge.
Key Insights and Ideas
- Interactive vs. Static: The core hypothesis is that the "pull" nature of a chatbot (asking specific questions) is more effective for learning than the "push" nature of a static guide (browsing a document).
- Quantifying Cognitive Load: By using the BDM bidding process, the study moves beyond simple "time-on-task" metrics to measure the subjective psychological cost of processing political information.
- LLM-in-the-Loop Evaluation: Interestingly, the researchers plan to use GPT-4o not just as the treatment (the chatbot) but also as the evaluator. They will use the model to score the quality, complexity, and persuasiveness of participants' open-ended reasoning regarding their vote choices.
- Heterogeneity Matters: The plan specifically looks for "heterogeneous treatment effects"—checking if the AI helps low-information voters more than high-information ones, or if it bridges gaps across different socio-economic backgrounds.
Key Results
As this is a Pre-Analysis Plan, quantitative results are not yet available.
The authors intend to estimate the treatment effect ($\tau$) using a linear OLS regression: $$Y_i = \tau T_i + \beta X_i + \epsilon_i$$ Where $Y$ represents outcomes like knowledge scores or turnout, and $T$ is the treatment indicator for BallotBot.
Limitations and Future Work
- Self-Reporting Bias: Turnout and voting choices are self-reported, which can be subject to social desirability bias (though the authors attempt to mitigate this through anonymity).
- The "Black Box" of LLMs: While the RAG approach limits hallucinations by grounding the AI in official text, the underlying GPT-4o model remains a proprietary system, which may limit the reproducibility of the exact "tone" or "logic" of the bot for other researchers.
- Experimenter Demand Effect: Participants might feel pressured to perform better because they know they are being tested on a "new" tool. The authors address this by emphasizing non-partisanship and ensuring high levels of participant privacy.
- Future Directions: The authors suggest that if BallotBot proves successful, it could serve as a blueprint for non-partisan, AI-driven civic engagement tools at the state or national level.
Summary updated