Back to the Future: On Potential Histories in NLP

Zeerak Talat, Anne LauscherarXiv 2022 · DOI · arXiv

Abstract

Current NLP practices often treat training data as an objective record, inadvertently reinforcing majoritarian biases. This paper proposes "History-Injection," a framework that treats dataset creation as an act of historical fiction to include marginalized voices. Using neopronouns as a case study, the authors demonstrate that fine-tuning models on these "potential histories" can shift internal representations to be more inclusive and representative of diverse lived realities.

Introduction

Machine learning models are typically trained on massive corpora that reflect dominant historical and contemporary narratives. The authors argue that this approach is fundamentally "conservative," treating the past as a singular, linear timeline. Consequently, minoritized linguistic practices—such as the use of neopronouns like xe/xem—are often dismissed as "noise" or outliers. This paper seeks to move NLP from merely reflecting existing data to actively constructing more equitable "potential histories."

Methodology

The researchers introduced "History-Injection," a process where a standard dataset is intentionally perturbed to create a "fiction" where marginalized language is the norm. They modified the Wikitext-103-v1 corpus by replacing traditional gendered pronouns (he, she, her, his) with corresponding neopronoun forms (xe, xem, xyr, xyrs, xemself). This creates a training environment where neopronouns are as common and grammatically integrated as traditional ones.

Two methods were tested for injecting this history into a Pre-trained Language Model (RoBERTa-large):

  1. Full Fine-tuning: Retraining the entire model on the fictionalized data.
  2. Adapter-based Injection: Inserting modular "adapter" layers, which is more computationally efficient and preserves the model's original broad knowledge while allowing for targeted ethical adjustments.

Vector Space Topology and Results

In standard models, neopronouns typically reside at the periphery of the vector space, far from the semantic center of "personhood" occupied by traditional pronouns. The History-Injection successfully shifted these representations.

PCA visualizations showing the movement of neopronouns from the periphery to the center of the pronoun cluster after history injection.

As shown in Figure 1, the original model (a) treats neopronouns as distant outliers. Full fine-tuning (b) aggressively moves xe and its variants into the same cluster as he and she. Adapter-based injection (c) achieves a similar, albeit slightly more moderate, normalization, effectively "centering" marginalized terms within the model's internal logic.

Quantitatively, the authors measured the cosine similarity between pronouns and the neutral anchor word "person" across all 24 layers of the model. In the baseline model, traditional pronouns were significantly closer to "person" than neopronouns. After full fine-tuning, xe became closer to "person" than traditional pronouns across nearly all layers. The adapter method also showed a significant shift toward the center, demonstrating that inclusivity can be achieved efficiently.

Heatmaps illustrating the change in pronoun representations across model layers for different fine-tuning methods.

Discussion and Ethics

The authors emphasize that data-making is an inherently political and creative act. By using modular adapters, researchers can "plug in" different ethical configurations or potential histories without retraining massive models from scratch. However, they also warn of "dual-use" risks: the same techniques used for inclusion could potentially be used to erase histories or reinforce harmful narratives. Furthermore, as language and social identities are fluid, these "potential histories" must be viewed as dynamic rather than static solutions.

Conclusion

By reframing NLP as a site of historical narrative-building rather than objective recording, we can intentionally design models that represent a broader range of human experiences. History-Injection offers a path toward normalizing marginalized language within the internal logic of large-scale language models, moving toward a more inclusive future for AI.

Summary updated

BibTeX
@misc{talat2022back,
  author = {Zeerak Talat and Anne Lauscher},
  title = {{Back to the Future: On Potential Histories in NLP}},
  year = {2022},
  eprint = {2210.06245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.06245},
}
Figure