When Parliamentary Language Models Count Women's Mentions

Researchers who study gender in politics increasingly measure it with language models. They count who gets mentioned by peers, who is addressed directly versus talked about, and whose speech is attributed to whom. A natural upgrade is to swap generic models for ones adapted to parliamentary text through continual pre-training: further training a general model on unlabelled proceedings so it learns the domain's vocabulary and style. This note argues that the upgrade carries a specific risk. Proceedings are dominated by a small number of senior, mostly male speakers. A model adapted on that text may get better on average at finding references to legislators while getting relatively worse at finding references to women. If so, the instrument would quietly change the size of the gender gap it is used to measure.

The proposal tests that risk directly and offers a cheap fix. We will build a gender-stratified benchmark of the hard cases in parliamentary mention detection. These are references that a name list cannot catch: titles such as "the honourable member" or "the minister", role descriptions, and direct address. We will then measure two things. First, whether domain-adapted encoders widen the gap between how often they find references to women and how often they find references to men. Second, whether that gap distorts a published disparity estimate, namely the finding that female Galician legislators receive less than half the peer mentions their male colleagues receive. The intervention is speaker-balanced continual pre-training. It caps each speaker's share of the adaptation text and evens out the gender mix, while keeping the total number of training tokens fixed.

Core Claim

Domain adaptation and measurement fairness are separate properties, and the field has studied only the first. Continual pre-training on raw proceedings improves accuracy on parliamentary text but leaves the gender gap in error rates unchanged or larger. Rebalancing the adaptation text by speaker narrows that gap at little or no cost to overall accuracy. The gap matters because it propagates into the substantive conclusions of gender-disparity studies.

Grounding in Prior Work

Two kinds of work motivate the proposal. The first shows that measurement pipelines fail unevenly by gender. Quantifying Gender Biases Towards Politicians on Reddit analysed 10 million comments and reported that its off-the-shelf entity linker reached only 50% accuracy on female politicians, which means the women's subsample was silently thinned. Bulgarian ParlaMint 4.0 corpus as a testset for Part-of-speech tagging and Named Entity Recognition showed an 8–9% drop in named-entity recognition when general Bulgarian models move to parliamentary text, but it never split errors by referent gender. The second kind supplies the target estimates. Gender, Speech, and Representation in the Galician Parliament found the passive-mention asymmetry using string matching and manual disambiguation. That study did not use learned models at all, so its counts cover mostly explicit name mentions. Distinguishing Address vs. Reference Mentions of Personal Names in Text showed that women are addressed directly more often than men and that direct address carries more toxicity. Its method probes generic BERT by masking a name and comparing the predicted probabilities of "you" versus "he"/"she", and it was tested only on social media and TED talks.

One line of work supplies the method. The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedings built XLM-R-parla by continually pre-training XLM-R-large on 27 parliaments, and UPPAM pre-trained an encoder for political actors using voting and co-sponsorship structure. Neither paper audits how its training text is distributed across speakers, and neither evaluates any task stratified by gender. We reuse the XLM-R-parla recipe and change only how the text is sampled.

Other work constrains the design. Arguments for training on isolated domain data, such as A Deep Fusion Model for Domain Adaptation in Phrase-based MT and MonoCoder, predict that keeping the adaptation corpus purely parliamentary helps. The participation skew documented in Greek proceedings by Investigating Political Ideologies through the Greek ParlaMint corpus and A Greek Parliament Proceedings Dataset for Computational Linguistics and Political Analysis suggests that purity comes at a cost. Within that pure domain text, fewer than 15% of Galician MPs deliver over half the speeches. The closest audit work, Identifying Algorithmic and Domain-Specific Bias in Parliamentary Debate Summarisation, measures bias by speaking order and party but not by gender. It also uses general models, so it says nothing about whether domain adaptation changes the bias.

Research Program

Benchmark. We will annotate a mention benchmark in three ParlaMint parliaments: Galician (ParlaMint-ES-GA), Greek, and Bulgarian. The three languages share grammatical gender but use different scripts. Annotators mark every reference to a sitting legislator that is not an exact name match. That covers titles, role descriptions ("the minister for health"), constituency references, pronouns that refer back to a legislator, and vocative address. For each reference, annotators record the legislator it points to and whether it is address or reference. Legislator gender comes from ParlaMint speaker metadata, so the annotators never label gender themselves.

We deliberately exclude exact name matches. In a parliament with a known membership list, matching names against the roster is close to solved, and a benchmark full of easy cases would hide the effect we are looking for. We will sample sessions so that references to women and to men are roughly equally frequent. Double annotation of a subset gives agreement figures; for comparison, the Facebook address/reference set reached Cohen's κ = 0.898.

Models. All encoders start from XLM-R-large. We compare four variants:

  • the base model with no adaptation;
  • the released XLM-R-parla;
  • our own raw continual pre-training on the three parliaments;
  • speaker-balanced continual pre-training on the same parliaments, with an identical token budget and number of steps.

The balanced sampler caps any single speaker's contribution at a fixed share of the corpus and reweights so that each gender contributes in proportion to its share of members, not its share of floor time. A second ablation caps per-speaker share but does not equalize gender. This separates the effect of removing domination by a few prolific speakers from the effect of the gender rebalancing itself.

Task heads. Each encoder is fine-tuned on the same labelled training split for two tasks: mention detection and linking to a legislator, and classification of address versus reference.

Propagation analysis. We run each pipeline over the full Galician proceedings and recompute the female-to-male ratio of passive mentions per legislator. That ratio is then compared with the same ratio computed on the gold-annotated sample.

Baselines

The do-nothing reference is roster-based string matching of the kind Vladu et al. used: legislators' surnames and full names matched against ParlaMint speaker metadata. By construction it scores near zero on our non-name benchmark. It still matters because it represents the published estimate, and the propagation analysis asks how far that estimate is from gold once indirect references are counted.

The strongest existing method is XLM-R-parla, fine-tuned on our training split with the same hyperparameter search as every other encoder (a fixed grid of learning rates and seeds, with model selection on overall dev F1 rather than gender-stratified F1). XLM-R-parla is the most extensively domain-adapted multilingual parliamentary encoder in the file. The only number reported for it is on sentiment: R² = 0.672 on the BCS test set versus 0.653 for XLM-R-large (ParlaSent). We found no reported result on Lacuna for gender-stratified mention detection with any parliamentary encoder. The nearest external figure for address versus reference is 85% accuracy from probing generic BERT on Facebook political mentions (Prabhakaran et al.). We will run that probe unchanged on our benchmark as a zero-shot reference point.

The key ablation is raw versus speaker-balanced continual pre-training at matched token budget. These two arms differ only in which documents are sampled. The proposal counts as beating XLM-R-parla and raw adaptation if two things hold. First, the absolute gap between female-referent and male-referent F1 is smaller across five seeds, with non-overlapping bootstrap intervals. Second, overall F1 stays within one point. To check that balancing does not erode general domain ability, we also report ParlaSent R² for every adapted encoder.

Why not just…?

Why not just match names against the membership roster, since parliaments are closed sets? For explicit names this works well, which is exactly why our benchmark excludes them. Gender-disparity claims about recognition depend on the indirect references that roster matching cannot see. The propagation analysis tests directly whether those references change the passive-mention ratio. If the gold ratio including indirect references matches the roster-only ratio, then roster matching is enough for this estimate, and we will say so.

Why not just fine-tune a general model on labelled in-domain data and skip pre-training? This is a real competitor: labelled data may matter more than unlabelled adaptation for a narrow task. The base XLM-R-large arm is exactly this baseline, fine-tuned on the same split. Männistö's finding that domain pre-training predicts downstream performance better than the choice of fine-tuning method (A Comparative Study of PEFT Methods for Python Code Generation) comes from code generation, so it does not settle the question here. If base-plus-fine-tuning matches the balanced encoder on both overall F1 and the gender gap, the conclusion becomes "skip adaptation", and that is still a useful negative result for practitioners.

Why not just correct the disparity estimate statistically, using per-gender error rates measured on a validation set? This is the most serious threat. Correction for measurement error can remove bias from an aggregate ratio without touching the model. We will apply it to the raw-adapted pipeline and compare the corrected ratio with gold. Correction has two weaknesses. It needs fresh gender-stratified labels for every parliament and time period, and it fixes aggregate counts but not the individual mentions that downstream analyses of framing and toxicity consume. If correction alone recovers the gold ratio across all three parliaments, the model-side intervention is unnecessary for aggregate estimates, and our main contribution becomes the error audit and the benchmark.

Why not just prompt a frontier LLM to annotate mentions? Cunningham et al. found that prompting with explicit fairness instructions did not remove representation disparities in parliamentary summaries (Identifying Algorithmic and Domain-Specific Bias in Parliamentary Debate Summarisation). Annotation is a different task, though, so we will include a single prompted open-weight model run on the test set and report its gender-stratified F1.

Decisive Checks

The first check is whether raw continual pre-training widens the female–male F1 gap relative to the unadapted base. If it does not in any of the three parliaments, the core worry is unfounded, and the speaker-balancing arm reduces to a robustness check.

The second check is whether, at matched tokens, speaker balancing narrows the gap while keeping both overall mention F1 and ParlaSent R² within one point. The cap-only ablation tells us whether any improvement comes from removing domination by prolific speakers or from equalizing gender.

The third check is propagation. For each pipeline, how far is the estimated passive-mention ratio from the gold ratio? This analysis also tests a prediction specific to these languages. Grammatical gender marks titles in Galician, Greek, and Bulgarian (for example a conselleira, "the minister" in feminine form). That morphology could make female references easier to detect, which would shrink the gap regardless of how the training text is sampled. Finding this would itself limit how far English-centred pipeline-bias results transfer to other languages.