Introduction
In modern representative democracies, parliamentary transcripts serve as the primary record of legislative discourse. However, the sheer volume and complexity of these records often create a "democratic deficit," where the public remains disconnected from the nuances of policy debates due to the time required to digest them. While Large Language Models (LLMs) offer a promising solution for automated summarisation, they introduce significant risks of distortion. Researchers at University College Dublin have developed a framework to identify and mitigate these risks, focusing on algorithmic and domain-specific biases that can compromise the integrity of the political record.
The Problem: Hidden Biases in Summarisation
Automated summaries are not neutral reflections of source text. The researchers identify two critical forms of bias that can transform a summary into a tool for misinformation:
- Algorithmic Bias (Positional): Known as the "lost-in-the-middle" phenomenon, models tend to prioritize information at the beginning and end of a long input. In a parliamentary context, this effectively silences speakers who contribute during the middle of a session.
- Domain-Specific Bias (Partisan): There is a risk that models may represent the arguments of certain political groups more faithfully than others, potentially due to the linguistic styles or the prevalence of certain ideologies in the models' pre-training data.
Crucially, traditional evaluation metrics like ROUGE or BERTScore are often "who-blind." They measure whether the general content is preserved but fail to detect if a specific argument has been incorrectly attributed to a different speaker or political party.
Methodology: A Hierarchical Framework
To address these issues, the study proposes a Hierarchical Summarisation Framework. Instead of feeding an entire debate transcript into a model at once, the process is split into two distinct stages:
- Step 1: Intervention Summaries. Each individual speech (intervention) is first summarised into a structured format. This structure captures the Headline, Issue, Position, Argument, Proposal, and specific Quotes.
- Step 2: Executive Summary. A "generator" function then aggregates these structured snippets into a final, coherent summary of the entire debate.
The Reconstructor Function ($f_{REC}$)
To quantify bias, the researchers introduced an innovative evaluation method: the Reconstructor Function. This involves using an LLM to attempt to "reconstruct" a speaker's original contribution using only the final summary as a source. If the reconstruction is high-quality (validated against the original speech using BERTScore), it indicates that the summary successfully captured and attributed that speaker’s specific views. This treats summarisation as a "lossy compression" task where fairness is measured by the remaining information signal for each participant.
Experimental Setup and Dataset
The framework was tested using translated debates from the 9th European Parliament (2019–2025). The dataset comprised:
- 93 full debates.
- 2,976 individual interventions.
- 719 unique speakers across 9 distinct political party groups.
The study compared four generator strategies: Default (concatenated text), Grouped (by subheadings), Hierarchical (independent sub-component processing), and Prompted (explicit fairness instructions).
Key Findings
Positional Bias is Model-Specific
The "lost-in-the-middle" effect varied significantly across models. Proprietary models like GPT-4 and Claude 3.7 Sonnet showed a preference for earlier speakers. Conversely, open-weight models such as Mistral-7B and Phi-4 tended to favor speakers at the very end of the debate. In the "Default" setup, fidelity for middle speakers often dropped to near zero for smaller models, meaning their contributions were entirely erased from the summary.
Hierarchical Processing as an Equalizer
The hierarchical approach proved to be a powerful "bias equalizer." By forcing the model to process each intervention separately before aggregation, the positional bias was significantly mitigated. This method produced the most information-dense summaries and ensured that speakers in the middle of the debate were represented with similar fidelity to those at the boundaries.
Partisan Representation Disparity
Even when controlling for speaking order, the models exhibited partisan preferences. Interventions from "The Left" party group were consistently represented more faithfully than those from the "Identity and Democracy" group. This suggests that LLMs may have internal "preferences" or that their training data better prepares them to summarise certain political rhetorics over others.
Conclusion and Future Directions
The research highlights that while LLMs are powerful tools for democratic transparency, they require structured frameworks to ensure fairness. The Reconstructor Function provides a new lens for evaluating fidelity beyond simple keyword matching, though the authors acknowledge the limitation of using one "black box" (an LLM) to evaluate another.
Future work aims to integrate these findings into public-facing legislative engagement platforms and investigate the underlying causes of partisan bias—whether it stems from pre-training data or specific linguistic styles used by different political factions.
Summary updated