Abstract
For over two decades, Natural Language Processing (NLP) systems have systematically struggled to correctly identify independent possessive pronouns such as "hers" and "theirs," often mislabeling them as nouns or adjectives while correctly identifying the masculine counterpart "his." This paper introduces a novel 7-step framework using Bayesian-based partially-synthetic data generation to detect and quantify this bias. Our findings reveal that modern transformer-based models like BERT do not merely reflect training data imbalances but significantly amplify them, creating a substantial hurdle for downstream tasks like co-reference resolution.
Introduction: The Independent Pronoun Oversight
In English, the word "his" serves a dual role as both a dependent possessive (e.g., "his car") and an independent possessive (e.g., "the car is his"). In contrast, "hers" and "theirs" are exclusively independent. Despite their common usage, major NLP parsers and language models from Google, AWS, and academia have historically failed to recognize this distinction. When a model cannot identify these words as pronouns, it fails to link them to their antecedents, leading to biased outputs in applications ranging from predictive text to automated translation.
Methodology
To systematically measure this bias, we propose a workflow that leverages contextual modeling and synthetic expansion to probe a model's internal associations.
The 7-Step Workflow
- Contextual Modeling: Utilize a pretrained model (e.g., BERT) as the baseline.
- Seed Dataset: Curate a small set of 50+ diverse syntactic positions for independent possessive pronouns (IDPs).
- Attribute Masking: Mask the "possessed" object in templates (e.g., "The [MASK] is hers").
- Bayesian Deep Learning: Apply Monte Carlo Dropout at inference time to generate a wide variety of candidate attributes (e.g., "house," "idea") that fit the syntactic context.
- Combinatorial Expansion: Combine generated attributes with original templates to create thousands of unique test sentences.
- Pronoun Prediction: Mask the pronoun (e.g., "The camera is [MASK]") and evaluate the model's predicted probabilities for "his," "hers," and "theirs."
- Ratio Calculation: Compare model prediction ratios against the actual frequency of these pairings in the training corpora (Wikipedia and BookCorpus).
Key Results and Bias Amplification
Our analysis demonstrates a phenomenon of bias amplification. If "his" appears four times more frequently than "hers" in training data, a neutral model might be expected to reflect that 4:1 ratio. However, we found that model preferences often reached 10x or 20x, indicating that the model architecture and training process exacerbate existing societal imbalances.
Quantitative Findings
- Universal Failure: Every major commercial and academic parser tested from the last 20 years consistently mislabeled "hers" and "theirs."
- Systematic Preference: For 103 out of 104 generated attributes, BERT preferred "his" over "hers" or "theirs."
- Statistical Significance: In 92 out of 104 cases, the model's bias toward "his" was at least three standard errors higher than the maximum frequency found in the training data.
- The "Ship" Exception: A notable outlier was the attribute "ship," where "hers" was predicted more frequently, likely reflecting the linguistic tradition of referring to maritime vessels with feminine pronouns.
Discussion and Limitations
While this framework provides a robust method for detecting bias, it does not inherently mitigate it. The study highlights that linguistic asymmetry acts as a trap; because "his" is used more diversely, models receive more "practice" with it, treating "hers" and "theirs" as rare or pathological cases.
Furthermore, technical hurdles such as tokenization play a role. BERT’s tokenizer often splits words like "themself" into sub-tokens ("them," "##s," "##elf"), creating a design bias before processing even begins. Future work should focus on systematic debiasing strategies, such as explicitly encoding grammatical categories (e.g., distinguishing his{DEP} from his{IND}) and addressing polysemy in pronouns like "mine," which can also function as a noun.
Summary updated