Underspecification in Language Modeling Tasks: A Causality-Informed Study of Gendered Pronoun Resolution

Emily McMilinAAAI 2024 · DOI

Introduction

Large Language Models (LLMs) often exhibit gender bias even when the input context appears neutral. This paper, presented at AAAI-24, investigates this phenomenon through the lens of underspecification. The authors argue that models resort to "last resort" spurious correlations when they lack sufficient information to make a principled decision.

Defining the Problem: Shortcut Learning vs. Underspecification

The study distinguishes between two primary failure modes in language models:

  1. Shortcut Learning: The model relies on an easy-to-learn feature (e.g., a gender-occupation stereotype) even when the intended logic is available in the prompt.
  2. Underspecification: The input context is fundamentally insufficient to determine the output (e.g., "In 1880, [MASK] was an adult").

In cases of underspecification, the model is forced to minimize its loss function by selecting a token, leading it to trigger "latent selection biases"—spurious correlations present in the training data that only manifest when no other cues are present.

Causal Framework and Methodology

The researchers employ Causal Directed Acyclic Graphs (DAGs) to model the relationships between input features ($X$), labels ($Y$), and selection bias ($S$).

Causal DAGs representing well-specified (a, c) and underspecified (b, d) scenarios for gendered pronoun resolution.

As shown in Figure 1, the presence of a disambiguating pronoun (like "his" in example a) creates a well-specified task. Without it (example b), the model must rely on the latent correlation between the occupation "surgeon" ($B$) and gender ($Y$). Similarly, examples (c) and (d) illustrate how temporal features ($W$) can drive predictions in the absence of explicit gender information.

Measuring Latent Correlations

By injecting "benign" features—such as specific years or geographic locations—into gender-neutral templates, the authors quantified previously unreported spurious correlations. For instance, models were found to shift their pronoun predictions based on the country or century mentioned, despite these being gender-neutral contexts.

The Task Specification Metric

The authors propose a Task Specification Metric to detect when a model is relying on bias. The intuition is that a well-specified task should be "time-invariant." If changing a benign feature (like shifting the year from 1880 to 2020) significantly alters the model's prediction, the task is likely underspecified. This black-box method requires no access to model weights or gradients.

Datasets and Evaluation

The study utilized two primary evaluation sets:

  • Masked Gender Challenge (MGC): A new dataset of gender-neutral templates like "In {year}, [MASK] was a {life stage}."
  • Extended Winogender Schema: A modified version of the standard Winogender benchmark, including "well-specified" versions where gender is explicitly stated to test the detection metric's ability to distinguish between "knowing" and "guessing."

Key Results and Insights

  • Detection Accuracy: The Task Specification Metric achieved a balanced accuracy of ~84% in identifying underspecified tasks for models like RoBERTa-large and GPT-3.5.
  • The Impact of Scale: Surprisingly, model size (from BERT-base to GPT-3.5) had little effect on the magnitude of these biases. Larger models are just as prone to these specific spurious correlations as smaller ones.
  • Training and RLHF: Models trained with diverse objectives (e.g., BART, UL2) showed smaller bias slopes. However, post-training techniques like Reinforcement Learning from Human Feedback (RLHF) were found to significantly alter the "bias profile," often increasing sensitivity to latent correlations.

Conclusion

This research provides a rigorous causal look at how underspecification induces bias in LLMs. By identifying these "methods of last resort," the authors offer a framework for detecting and potentially mitigating the reliance on spurious correlations in ambiguous contexts.

Code and demos for this paper are available at https://github.com/2dot71mily/uspec.

Summary updated

BibTeX
@inproceedings{mcmilin2024underspecification,
  author = {Emily McMilin},
  title = {{Underspecification in Language Modeling Tasks: A Causality-Informed Study of Gendered Pronoun Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29842},
}
Figure