GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German

Fabian Mewes, Anne Lauscher, Vagrant GautamarXiv 2026 · arXiv

TL;DR

The paper introduces GRUFF, the first large-scale benchmark for evaluating pronoun fidelity—the ability of language models to correctly track and reuse specified pronouns—in German. The authors find that while German's rich grammatical gender system helps models resist single distracting entities better than in English, LLMs fail completely at using gender-inclusive neopronouns unless they are explicitly provided in the context.

What Problem Does This Paper Address?

When a language model processes a long text, it needs to remember which pronoun belongs to which person, even when other people (distractors) are mentioned in between. This capability is called pronoun fidelity.

While benchmarks exist for English (like RUFF), English has very limited grammatical gender. In contrast, German is morphologically rich: articles, nouns, and pronouns must all agree in gender (masculine, feminine, or neutral) and grammatical case (nominative, accusative, dative, genitive). Furthermore, the German queer and non-binary communities have introduced various gender-inclusive noun forms (like the Sternchen: Tierarzt*in) and neopronouns (like xier or en). This paper investigates how well current LLMs navigate pronoun tracking, grammatical agreement, and occupational biases within German's complex linguistic landscape.

Methods

The authors created a framework to generate controlled linguistic templates in German. Each task instance consists of:

  1. An introduction sentence: Establishes an entity (e.g., Die Tierärztin / the female vet) and its pronoun (sie / she).
  2. Distractor sentences: Introduce 0 to 5 other entities with different genders/pronouns to confuse the model.
  3. A task sentence: A sentence with a blank gap where the model must fill in the correct pronoun referencing the original entity.

The dataset generation pipeline is visualized below:

Figure 2: Overview of the creation of each task instance in GRUFF, which involves sampling from occupation- participant pairs (Section 4.1), article-noun-pronoun agreement systems (Tables 1 and 2), and context and task templates (Section 4.2), and assembling them together (Section 4.3).

To evaluate the models, the authors used a forced-choice setting. Instead of letting the model freely generate text, they calculated the average sequence probability (log-likelihood) for four pronoun options, selecting the highest-probability option as the model's answer. They tested four encoder-only models (like GBERT and mBERT) and three decoder-only models (like SauerkrautLM and LLaMA-3).

Datasets

The paper introduces GRUFF (German Pronoun Fidelity dataset).

  • It features 60 occupation-participant pairs translated and adapted to German.
  • It covers four noun agreement systems (Masculine, Feminine, De-e neosystem, and the gender-inclusive Sternchen).
  • It spans four pronoun sets (er, sie, en, xier) across four grammatical cases (Nominative, Accusative, Dative, Genitive).
  • Combining these permutations yields over 7 million unique evaluation instances, which were downsampled into controlled subsets for testing.

Key Insights and Ideas

  • Zero-Shot Neopronoun Blindness: Without explicit context, LLMs show near-perfect grammatical agreement for traditional masculine and feminine nouns. However, they show near-zero agreement for neopronouns (en and xier). They simply do not inherently know which neopronoun maps to a gender-neutral noun variant.
  • The Tokenization Trap for en: Even with explicit context, the neopronoun en performs drastically worse than xier. The authors hypothesize this is due to tokenization: "en" is a ubiquitous substring in German (appearing in verb infinitives like ess**en and prefixes like ent-), causing severe embedding fragmentation, whereas xier is linguistically unique.
  • Morphology as a Shield Against Distractors: In English benchmarks, a single distractor sentence causes model accuracy to plummet. In German, encoder-only models are surprisingly robust against one distractor because the explicit gender markings on German articles and nouns provide a stronger, redundant guiding signal. However, adding two or more distractors breaks this robustness.
  • Uncorrelated Biases: Unlike English, where occupational stereotypes strongly bias models across the board, German occupational biases are poorly correlated across different grammatical cases and different model architectures.

Key Results

  • Grammatical Agreement: With no context, traditional pronouns score ~100% accuracy, while neopronouns score ~0%. Providing an explicit introductory context brings xier up to 66.74% accuracy, but en lags behind at just 23.49%.
  • Robustness: The vast majority of errors occur because models default to repeating the most recent distractor pronoun. Encoder-only models handle single distractors well but experience their sharpest performance drop when a second distractor is introduced.

Limitations and Future Work

  • Synthetic Data: The dataset relies on researcher-constructed templates rather than natural, fluid discourse.
  • Evaluation Constraints: The forced-choice probability setup does not capture how models behave during free-form text generation or within conversational chat interfaces.
  • Lack of Social Consensus: There is currently no single consensus in Germany on which gender-neutral grammar system is standard; GRUFF only tests a subset of popular community recommendations (De-e and Sternchen).
  • Future Directions: The authors call for deeper investigations into how German tokenizers handle neopronouns and advocate for closing the quality-of-service gap for non-binary language in multilingual NLP models.

Summary updated

Abstract

Third-person singular pronouns have long been used to study stereotypical biases in language models and to test their abilities to reason about reference. More recently, the interplay between reasoning and bias has been investigated with the task of pronoun fidelity, which assesses models' abilities to correctly reuse a previously-specified pronoun for a discourse entity, independent of other potentially distracting discourse entities mentioned in between. However, such research focuses on English, which is a language with limited grammatical gender and almost no gender agreement. In this paper we contribute a novel, large-scale dataset, GRUFF, to measure pronoun fidelity in German, covering four different gender agreement systems in nouns, and four sets of pronouns. With this dataset, we show that LLMs show strong grammatical agreement for masculine and feminine entities in the absence of explicit context, but not for neopronouns xier and en. Models are generally not robust to distractors, but encoder-only models are more robust in German than in English, reflecting the importance of grammatical gender. Finally, we show that occupational stereotypes in this context are poorly correlated across grammatical cases, and across most models, except ones with closely related architectures. We release all code and data to encourage further work on gender-inclusive language and referential reasoning in German.
BibTeX
@misc{mewes2026gruff,
  author = {Fabian Mewes and Anne Lauscher and Vagrant Gautam},
  title = {{GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German}},
  year = {2026},
  eprint = {2605.30214},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.30214},
}
Figure