Mind the Gap: Gender-based Differences in Occupational Embeddings

Olga Kononykhina, Anna‐Carolina Haensch, Frauke Kreuter2025 · DOI

TL;DR

Researchers found that state-of-the-art Large Language Models (LLMs) struggle to recognize that masculine and feminine job titles in German (like Autor and Autorin) refer to the same occupation. This linguistic "blind spot" causes models to suggest different occupational codes for identical jobs 54% to 82% of the time, potentially skewing labor market statistics and policy decisions.

What Problem Does This Paper Address?

Occupational coding is the process of mapping free-text job descriptions (e.g., "I write software for cars") to standardized categories used by governments and researchers to track employment trends, wage gaps, and public health. Historically, this was a manual, labor-intensive task. Today, researchers are turning to LLM-based embeddings to automate the process by measuring the semantic similarity between a person's self-reported title and an official taxonomy.

However, many languages—including German—are morphologically rich. A male author is an Autor, while a female author is an Autorin. While these terms are semantically identical in terms of the work performed, they are lexically different. This paper investigates whether embedding models are "fooled" by these gendered suffixes, leading to inconsistent or biased classification in social science research.

Methods

The authors evaluated five prominent multilingual embedding models:

  • MiniLM-L12-v2
  • Alibaba NLP gte-multilingual-base
  • Gemini’s text-embedding-004
  • GPT text-embedding-3-large
  • IBM’s granite-embedding-278m-multilingual

The team used a "retrieval" approach: they took a job title, converted it into a vector (embedding), and found the top 5 most similar categories from the official German Classification of Occupations (KldB 2010). They tested two main metrics:

  1. Gender-based divergence: How often do the masculine and feminine versions of the same title result in different Top-5 suggestions?
  2. Top-5 accuracy: How often does the "gold standard" (the code assigned by professional human coders) actually appear in the model's Top-5 list?

Datasets

The study utilized two primary data sources:

  • KldB 2010: The official German taxonomy containing 1,286 occupational categories, each with descriptions, typical tasks, and example titles.
  • INFAS 2019 Survey: A representative German survey of 1,415 adults. From this, the authors extracted 276 gendered responses, which were deduplicated into 39 distinct masculine-feminine job title pairs (78 total observations) for the final evaluation.

Key Insights and Ideas

  • Surface Form vs. Semantic Meaning: The models often prioritize the "lexical surface form" (how the word is spelled) over its functional meaning. For example, Autor (male) might be grouped with "Composer," while Autorin (female) gets grouped with "Editor."
  • More Context Isn't a Cure-all: One might assume that adding detailed job descriptions or task lists to the reference data would help the model focus on the work rather than the title. Surprisingly, the researchers found that adding more text did not systematically reduce gender divergence.
  • Model "Personalities" in Bias: Bias isn't uniform across AI. GPT showed a distinct "female bias," being significantly more accurate at identifying the correct code for feminine titles than masculine ones (a gap of up to 18 percentage points). IBM was less accurate overall but much more balanced between genders.
  • The "Generic Masculine" Problem: In German, the masculine form has traditionally been used as a "generic" term for all genders. The study suggests that LLMs may be internalizing these historical linguistic patterns in ways that produce inconsistent results for modern, gender-specific inputs.

Key Results

  • High Divergence: Across all models and configurations, 54% to 82% of gendered job title pairs received at least one different suggestion in their Top-5 list.
  • Accuracy Gaps:
    • GPT was the most accurate (62% inclusion of the correct code) but showed the highest gender disparity (+18 pp favoring females in some settings).
  • IBM hovered around 51% accuracy but remained broadly gender-neutral.
  • Alibaba, Gemini, and MiniLM all achieved roughly 50% accuracy, with smaller, fluctuating gaps that the authors attribute more to "retrieval noise" than systematic bias.

Limitations and Future Work

The authors acknowledge several constraints:

  • Sample Size: The evaluation was limited to 39 pairs found in the survey data. Expanding this to the full 1,300+ categories in the KldB would provide a more comprehensive picture.
  • Binary Focus: The study focused exclusively on binary (masculine/feminine) forms, leaving out gender-neutral alternatives (like Lehrkraft instead of Lehrer) or inclusive forms (like Lehrerin*).
  • Methodological Scope: The study focused on cosine similarity of embeddings. Other methods, like few-shot prompting or "LLM-as-judge" setups, might yield different results.

The paper concludes that for automated occupational coding to be "fair," future systems must be explicitly trained or fine-tuned to recognize morphological variations as semantically equivalent, ensuring that a person's gender doesn't change how their contribution to the labor market is recorded.

Summary updated

Abstract

Large Language Models (LLMs) offer promising alternatives to traditional occupational coding approaches in survey research.Using a German dataset, we examine the extent to which LLM-based occupational coding differs by gender.Our findings reveal systematic disparities: gendered job titles (e.g., "Autor" vs. "Autorin", meaning "male author" vs. "female author") frequently result in diverging occupation codes, even when semantically identical.Across all models, 54%-82% of gendered inputs obtain different Top-5 suggestions.The practical impact, however, depends on the model.GPT includes the correct code most often (62%) but demonstrates female bias (up to +18 pp).IBM is less accurate (51%) but largely balanced.Alibaba, Gemini, and MiniLM achieve about 50% correct-code inclusion, and their small (< 10 pp) and direction-flipping gaps could indicate a sampling noise rather than gender bias.We discuss these findings in the context of fairness and reproducibility in NLP applications for social data.
BibTeX
@misc{kononykhina2025mind,
  author = {Olga Kononykhina and Anna‐Carolina Haensch and Frauke Kreuter},
  title = {{Mind the Gap: Gender-based Differences in Occupational Embeddings}},
  year = {2025},
  doi = {10.18653/v1/2025.gebnlp-1.7},
}
Figure