Generating Gender Augmented Data for NLP

Nishtha Jain, Maja Popović, Declan Groves, Eva VanmassenhoveACL 2021

TL;DR

This paper tackles gender bias in NLP by proposing an automatic method to generate gender-balanced conversational sentences, particularly for gender-inflected languages like Spanish. They achieve this by training a Neural Machine Translation (NMT) system to "translate" sentences into their gender-alternatives, creating a valuable resource for training less biased NLP models.

What Problem Does This Paper Address?

Natural Language Processing (NLP) systems, especially those dealing with conversational text, often exhibit gender bias. This bias can manifest in various ways, such as associating certain professions or adjectives with specific genders, or through an imbalance in how grammatical genders are used. A significant challenge arises when sentences don't explicitly specify the gender of a referent, forcing NLP models to rely on learned patterns that often reflect societal biases. This can lead to systems perpetuating or even amplifying these biases. Current solutions are often limited to specific phenomena (like professions) or require extensive manual effort. The authors aim to create a more generalizable and automatic approach to generate gender variants for short conversational sentences, thereby fostering inclusivity and enabling the creation of more balanced training data.

Methods

The core of their approach involves a two-stage process:

  1. Data Generation: They first create a large, gender-parallel Spanish dataset. This involves:
  2. Identifying Re-genderable Sentences: Using Part-of-Speech (POS) sequences, they identify sentences in Spanish that contain words (adjectives, past participles, clitic pronouns, demonstrative pronouns) that can be grammatically altered to express a different gender. They carefully define rules to distinguish between sentences where gender agreement is with a referent (and thus should not be changed) and those where gender can be altered.
  3. Applying Rules: For identified re-genderable sentences, they apply a rule-based approach to generate the alternative gender variant. This process is applied in both directions (masculine-to-feminine and feminine-to-masculine).
  4. Including Neutral Sentences: Crucially, they also include sentences that are not re-genderable (neutral) in their training data. This teaches the model when to generate alternatives and when to leave sentences unchanged.
  5. Neural Rewriter (NMT): They train a Neural Machine Translation (NMT) system (specifically, using the Sockeye implementation of the Transformer architecture) to act as a "gender rewriter." The NMT system learns to translate original sentences into their gender-alternatives. They experiment with two setups: a standard NMT system and an NMT system augmented with "neutral" (N) or "re-genderable" (G) tags.

Datasets

  • OpenSubtitles: A large corpus of Spanish subtitles was used as the primary source for extracting sentences. After filtering and POS tagging, over 1 million re-genderable sentences were identified. This data was used to create a training corpus of approximately 2.2 million gender-parallel sentences.
  • Industry Partner Data: A dataset of around 8,000 sentences, already provided with alternative gender versions, was used for initial development and testing. An additional 22,000 sentences from the same partner were used for testing, split into "structured" and "unstructured" sets to evaluate the robustness of their rule-based identification.

Key Insights and Ideas

  • The Power of Rules for Data Generation: Instead of relying solely on existing, often biased, data, the authors demonstrate that linguistic rules can be effectively used to create balanced, gender-parallel data. This is a significant step towards overcoming data scarcity.
  • NMT as a Gender Rewriter: Framing the problem as a translation task for NMT is a novel and effective way to automate the generation of gender variants, moving beyond simple word-swapping.
  • The Importance of Neutral Sentences: Including neutral sentences in the training data is critical. It teaches the NMT model to distinguish between sentences that should be altered and those that should not, preventing unwanted changes.
  • Tagging Improves Performance: Explicitly tagging sentences as "neutral" or "re-genderable" (either manually or via a classifier) significantly improves the NMT rewriter's performance, especially for unstructured sentences.

Key Results

  • The NMT rewriter achieved promising results, with error rates significantly lower on "structured" test sets (where their rules are applicable) compared to "unstructured" ones.
  • For "structured" test sets, the NMT rewriter with manual tags achieved an error rate of 4.5% (count of incorrectly converted words per sentence) and 1.03% (per word).
  • The gender classifier, trained on a small dataset, showed promising results, achieving 82% overall accuracy on an industry test set. It demonstrated high recall for re-genderable sentences, meaning it was good at identifying sentences that need alteration, though precision for neutral sentences could be improved.
  • Qualitative analysis showed the NMT rewriter generally targets specific words rather than rewriting entire segments, which is desirable. The most frequent errors were leaving re-genderable sentences unchanged or incorrectly re-gendering neutral sentences.

Limitations and Future Work

  • Binary Gender: The current approach is limited to generating binary gender alternatives (masculine/feminine) and does not address non-binary gender.
  • Short Sentences: The rule-based identification of re-genderable sentences is primarily designed for short sentences with at most one referent.
  • Scalability of Rules: While the NMT rewriter aims for scalability, the initial rule-based identification is language-specific (Spanish) and might require significant adaptation for other languages.
  • Classifier Performance: The gender classifier, while promising, could be improved with larger annotated training datasets.
  • Future Work: The authors plan to extend their approach to more complex sentence structures and different languages. They also suggest exploring alternative NMT architectures (e.g., character-level NMT) and incorporating more morphological information and word embeddings for improved classifier performance.

Summary updated

BibTeX
@inproceedings{jain2021generating,
  author = {Nishtha Jain and Maja Popović and Declan Groves and Eva Vanmassenhove},
  title = {{Generating Gender Augmented Data for NLP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure