Llama meets EU: Investigating the European political spectrum through the lens of LLMs

Ilias Chalkidis, Stephanie Brandl

TL;DR

This paper investigates the political leanings of Large Language Models (LLMs) within the complex European political landscape, moving beyond the US-centric two-party system. By fine-tuning Llama Chat on European Parliament debates and evaluating its responses to political questionnaires, the researchers demonstrate that LLMs can be aligned with specific political ideologies, offering a new tool for political science research.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly powerful tools, but concerns about their inherent political biases and their potential to influence downstream tasks are growing. While previous research has explored these biases, it has largely focused on the simplified US political context (Democrats vs. Republicans) and often used older LLM architectures. This paper addresses the gap by examining LLM political leanings within the more diverse and multi-party European Union (EU) context. It also explores whether LLMs can not only reflect but also reason about political stances and if they can be adapted to align with specific political parties.

Methods

The researchers employed a multi-pronged approach:

  • Contextualized Auditing: They used the EUANDI questionnaire, a benchmark designed to gauge political stances, to probe Llama Chat. This involved three settings:
  • Setting A: Providing the LLM with a national party's name and asking for its stance on a statement. This tests the LLM's inherent knowledge of party positions.
  • Setting B: Providing the LLM with a party's justification for a statement and asking it to predict agreement. This tests the LLM's ability to reason about political arguments.
  • Setting C: Providing a party's justification and asking the LLM to identify the party. This combines knowledge and reasoning.
  • Political Adaptation/Alignment: They fine-tuned the Llama Chat model using Low-Rank Adaptation (LoRA) on speeches from the EU Debates Corpus. This process aimed to align the LLM's responses with the specific ideologies of various euro-parties (e.g., EPP, S&D, GUE/NGL, Greens/EFA, ID).
  • "Jailbreaking" Prompts: To overcome Llama Chat's inherent safety restrictions that prevent it from expressing opinions, the researchers used specially crafted prompts to elicit responses on political topics.

Datasets

  • EU Debates Corpus: A newly released corpus of approximately 87,000 parliamentary speeches from the European Parliament (2009-2023). Speeches were machine-translated into English where necessary.
  • EUANDI Questionnaire: A public questionnaire used before the 2019 EU elections, containing 22 political statements across 7 thematic categories (e.g., Liberal Society, Environmental Protection, EU Integration). The dataset includes national parties' expected answers and justifications.

Key Insights and Ideas

  • LLMs Have Political Knowledge: Even the baseline Llama Chat model demonstrates a substantial understanding of the political leanings of various European parties.
  • Reasoning Capabilities: LLMs can reason about political statements and predict inclinations based on provided justifications, suggesting they can process nuanced political arguments.
  • Adaptability is Key: Fine-tuning LLMs on specific party speeches significantly shifts their political alignment, making them "data-driven mirrors" of those parties' ideologies.
  • "Big Tent" Parties are Harder to Align: Models align more effectively with parties that have a consistent ideology (e.g., Greens, GUE/NGL, ID) compared to "big tent" parties (e.g., EPP, S&D) that encompass a broader spectrum of views. This suggests that the internal diversity of a party impacts how well an LLM can capture its essence.
  • Justifications Trump Names: The model's accuracy in predicting stances is generally higher when provided with a party's justification rather than just its name, highlighting the importance of contextual reasoning.

Key Results

  • Contextualized Auditing:
    • In Setting A (party name context), Llama Chat achieved an average accuracy of 74.3% across EU states and parties.
  • In Setting B (party justification context), accuracy improved to an average of 76.8%, demonstrating the LLM's ability to leverage reasoning.
  • Setting C (identifying party from justification) showed varying but generally good performance, with the model often correctly identifying the party or politically similar ones.
  • Political Adaptation:
    • Fine-tuned models showed a clear shift in their political compass, aligning with the target euro-parties.
  • Alignment was more successful for parties with consistent ideologies. For example, the adapted Greens/EFA and GUE/NGL models closely mirrored their respective party stances, while EPP and S&D showed more divergence.
  • The study found that higher data availability for a party did not always correlate with better model alignment.

Limitations and Future Work

  • Model Size: The study was limited to a 13-billion-parameter Llama Chat model due to computational constraints. Future work will explore larger models like Mistral AI's Mixtral.
  • English-Only: The use of English-only models and machine-translated speeches introduces potential inaccuracies and biases. Future work aims to use multilingual models and analyze original language debates.
  • Option/Justification Misalignment: The model sometimes selected an answer option that contradicted its own justification. This highlights the need for more robust evaluation methods, possibly involving Chain-of-Thought prompting or more advanced LLMs.
  • Time-Frames: The study used a mix of historical debate data and more recent questionnaire data, which could lead to misalignment due to parties evolving over time. Future research will examine temporal drifts in political leanings.
  • Annotation Bias: Manual annotations were used, which are inherently subjective. Broader annotation pools are suggested to mitigate this.
  • Limited Data Coverage: The study focused on a subset of parties. Analyzing all parties would provide a more comprehensive understanding of general trends.
  • Data Skewness: The EU Debates dataset has uneven coverage of thematic areas, potentially impacting fine-tuning on topics like environment or immigration. Future work may involve balancing data across topics.
  • Ethical Considerations: The adaptation of LLMs to specific political parties raises ethical concerns, which the authors acknowledge and aim to address through responsible research and controlled model releases. Future work will focus on using LLMs to aid political science research and inform citizens.

Summary updated

Abstract

Instruction-finetuned Large Language Models inherit clear political leanings that have been shown to influence downstream task performance. We expand this line of research beyond the two-party system in the US and audit Llama Chat in the context of EU politics in various settings to analyze the model's political knowledge and its ability to reason in context. We adapt, i.e., further fine-tune, Llama Chat on speeches of individual euro-parties from debates in the European Parliament to reevaluate its political leaning based on the EUandI questionnaire. Llama Chat shows considerable knowledge of national parties' positions and is capable of reasoning in context. The adapted, party-specific, models are substantially re-aligned towards respective positions which we see as a starting point for using chat-based LLMs as data-driven conversational engines to assist research in political science.
BibTeX
@inproceedings{chalkidis2024llama,
  author = {Ilias Chalkidis and Stephanie Brandl},
  title = {{Llama meets EU: Investigating the European political spectrum through the lens of LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.40},
}
Figure