Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis

Tanusree Sharma, Yujin Potter, Zachary Kilhoffer, Yun Huang, Dawn Song, Yang Wang

TL;DR

This paper explores how to govern large language models (LLMs) that analyze political videos by using a novel decentralized, participatory approach called Inclusive.AI. They found that different governance mechanisms, like voting methods, significantly impact how users want LLMs to interpret political content, highlighting the need for democratic input in AI development.

What Problem Does This Paper Address?

The increasing use of AI, particularly LLMs, in analyzing sensitive content like political videos raises concerns about bias, transparency, and alignment with public values. Traditional methods for gathering user feedback are often insufficient for ongoing deliberation and capturing diverse perspectives. This paper tackles the challenge of how to effectively govern AI systems, specifically multimodal LLMs used for political video analysis, by incorporating public input in a structured and democratic way. The goal is to ensure AI behavior aligns with societal expectations and values, especially in politically charged domains where misinterpretations can have significant consequences.

Methods

The researchers employed a two-step study:

  1. Expert Interviews: 10 journalists and media professionals were interviewed to establish a baseline understanding of how experts interpret political videos. This helped in selecting a suitable video for the experiment and informing the study's methodology.
  2. Large-Scale User Experiment with Inclusive.AI: 114 US internet users participated in a study using Inclusive.AI, a platform designed for democratic decision-making through Decentralized Autonomous Organization (DAO) mechanisms. The experiment involved:
  3. Human-AI Deliberation: Users interacted with an AI (GPT-4) to discuss its interpretation of a political video, clarifying their values and expectations.
  4. Group Deliberation: Users engaged in discussions with each other to co-validate their values and perspectives.
  5. DAO Governance Voting: Users participated in voting on different LLM improvement options using various governance mechanisms.

The study specifically investigated how different voting methods (quadratic vs. weighted voting) and voting power distributions (equal vs. 20/80 split) influenced user decisions regarding LLM behavior.

Datasets

The study used a video from the 2020 US presidential debate as a seed case for the experiment. This video was selected based on expert interviews and criteria including relevance to current events, political context, depth, and authenticity. No new benchmark datasets were introduced; rather, a real-world political video was used for analysis and deliberation.

Key Insights and Ideas

  • Expert vs. Public Priorities: Experts focused on emotion and narrative in video interpretation, while the general public prioritized factual clarity, objectivity, and emotional neutrality. This highlights a crucial divergence in how different groups perceive the ideal AI output for political content.
  • DAO as a Governance Tool: The study demonstrates the potential of DAOs and participatory AI governance to facilitate broader public engagement in AI development, moving beyond traditional top-down approaches.
  • Voting Mechanisms Matter: The choice of voting method and power distribution significantly shapes user decisions and perceptions of democratic quality. Quadratic voting, in particular, was found to reinforce perceptions of liberal democracy and political equality.
  • The Need for Nuance in AI Interpretation: Users expressed a strong preference for LLMs to provide more specific facts and analyze emotions/sentiment, rather than just keeping the current model.
  • Skepticism about Implementation: Despite positive perceptions of the governance process, participants expressed skepticism about whether their voting outcomes would actually be implemented in LLM models, suggesting a need for government-level oversight.

Key Results

  • User Preferences: Participants strongly preferred LLM improvements that involved "providing more specific facts" (Choice 2), followed by "analyzing speakers' emotions and sentiment" (Choice 4).
  • Voting Method Impact: Quadratic voting, especially with equal power distribution, was perceived as more democratic and fair, reducing concerns about unexpected outcomes compared to weighted voting.
  • Political Affiliation Differences: Republicans were less likely to prefer "more specific facts" and more likely to favor integrating a user feedback loop compared to Democrats.
  • Perception of Governance: Participants generally rated the voting process positively across different mechanisms, with quadratic voting and equal power distribution enhancing trust and perceptions of fairness.
  • Correlation with Democracy Values: Participants who found LLM personally relevant viewed the DAO voting process as more participatory. Perceptions of deliberative democracy were strongly linked to trust in AI and perceived AI risks.

Limitations and Future Work

Limitations:

  • Generalizability: The study involved US residents, and the findings might not directly translate to other cultural or political contexts.
  • Implementation Skepticism: Participants' doubts about the actual implementation of their voting decisions highlight a gap between participatory processes and real-world AI development.
  • Scope of Political Content: The study focused on a single presidential debate video, and further research is needed to explore a wider range of political content.

Future Work:

  • Government-Level Guidelines: The authors suggest the necessity of government-level guidelines to ensure system compliance and evidence through future audits.
  • Broader Application: They envision integrating Inclusive.AI as a plugin within existing LLM applications (like ChatGPT) to allow users to challenge, deliberate, and vote on AI-generated content across various domains (politics, health, education).
  • Exploring Other Mechanisms: Future work could explore alternative governance mechanisms, such as sortition-based DAOs, to prevent power concentration and ensure unbiased participant selection.
  • Evaluating Deliberative Refinement: The authors plan to evaluate Inclusive.AI's deliberative refinement model against other approaches like top-down moderation, non-deliberative crowd voting, and expert-only audits.

Summary updated

Abstract

How AI models should deal with political topics has been discussed, but it remains challenging and requires better governance. This paper examines the governance of large language models through individual and collective deliberation, focusing on politically sensitive videos. We conducted a two-step study: interviews with 10 journalists established a baseline understanding of expert video interpretation; 114 individuals through deliberation using Inclusive.AI, a platform that facilitates democratic decision-making through decentralized autonomous organization (DAO) mechanisms. Our findings reveal distinct differences in interpretative priorities: while experts emphasized emotion and narrative, general public prioritized factual clarity, objectivity, and emotional neutrality. Furthermore, we examined how different governance mechanisms - quadratic vs. weighted voting and equal vs. 20/80 voting power - shape users' decision-making regarding AI behavior. Results indicate that voting methods significantly influence outcomes, with quadratic voting reinforcing perceptions of liberal democracy and political equality. Our study underscores the necessity of selecting appropriate governance mechanisms to better capture user perspectives and suggests decentralized AI governance as a potential way to facilitate broader public engagement in AI development, ensuring that varied perspectives meaningfully inform design decisions.
BibTeX
@inproceedings{sharma2025aligning,
  author = {Tanusree Sharma and Yujin Potter and Zachary Kilhoffer and Yun Huang and Dawn Song and Yang Wang},
  title = {{Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aies.v8i3.36720},
}
Figure