Predicting and Explaining French Grammatical Gender

Saumya Sahai, Dravyansh SharmaNAACL 2021

TL;DR

This paper tackles the challenge of predicting and explaining grammatical gender in French, a complex linguistic phenomenon. They introduce a novel approach that combines high-accuracy neural models with a global surrogate explanation method, allowing for tunable explanation complexity and providing insights into the linguistic factors driving gender assignment.

What Problem Does This Paper Address?

Grammatical gender, a feature of many languages, assigns a gender (masculine, feminine, or sometimes neuter) to nouns, influencing agreement with other words in a sentence. This can be a significant hurdle for language learners, especially those whose native languages lack grammatical gender. While traditional rule-based systems exist, they are often complex and difficult to maintain. Computational approaches offer scalability but often act as "black boxes," making it hard to understand why a prediction is made. This paper aims to bridge this gap by developing accurate predictive models for French grammatical gender that are also interpretable, offering explanations that can be understood by humans, including language learners.

Methods

The authors propose a two-pronged approach:

  • Predictive Models: They experiment with several models, including:
  • Semantic Models (SEM): Utilizing word definitions processed by bidirectional LSTMs or XLM-R embeddings.
  • Phonological Model (PHON): Employing n-grams of phoneme sequences.
  • Orthographic Model (ORTH): Using n-grams of word spellings.
  • Combined Models: Integrating features from orthography, semantics, and phonology.
  • Explainability Framework: This is the core novelty. They introduce a global surrogate based approach that maps the feature embeddings learned by the black-box neural models to an auxiliary space of explanations. This allows for:
  • Tunable Explanation Complexity: Auxiliary attributes can be designed to match the user's understanding level (e.g., a language learner).
  • Local Explanations: Extending the LIME approach to train local decision trees on nearest neighbors for neighborhood-specific explanations.
  • Global Explanations: Using decision trees on interpretable features to predict the black-box model's output, aiming for explanations that are faithful to the model's decision-making process but also intuitive.

Datasets

The primary dataset used is derived from Dbnary, a Wiktionary-based lexical database. French words, their definitions, and phonetic representations were extracted. The dataset was filtered to include only nouns with a single gender assignment. The resulting dataset contains 124,803 words, split into 90-10-10 for training, validation, and testing. The class distribution is balanced, with 58% masculine and 42% feminine words.

Key Insights and Ideas

  • Orthography is King (for French): The study reveals that word orthography alone is a surprisingly strong predictor of grammatical gender in French, achieving high accuracy.
  • Semantics Adds Value: While orthography is dominant, incorporating semantic information from word definitions further improves prediction accuracy, suggesting that meaning plays a role beyond just form.
  • Phonology's Limited Role: Phonological features, surprisingly, did not significantly improve prediction accuracy when combined with orthography and semantics. This is attributed to the fact that French orthography often disambiguates pronunciation, and may carry more etymological information than spoken forms.
  • Explainability Trade-offs: There's a clear trade-off between model fidelity (accuracy) and interpretability (simplicity of explanation). Larger decision trees lead to higher accuracy but are harder to understand.
  • Feminine Gender is More "Rule-Bound": Explanations for feminine nouns tend to be more interpretable (shorter decision tree paths) than for masculine nouns. This aligns with the observation that masculine gender often acts as a default category with more complex exception rules.
  • Auxiliary Features for Tunable Explanations: The concept of using auxiliary attributes to create explanations at different levels of complexity is a powerful idea, especially for applications like language learning.

Key Results

  • The best orthographic model achieved an accuracy of 92.5%.
  • The best semantic model alone achieved 77.23%.
  • Combining orthographic and semantic features yielded the highest accuracy at 94.01%.
  • The study demonstrates that their explainability approach can provide insights into gender assignment, with decision trees highlighting key orthographic and semantic features distinguishing gender. For instance, clusters of words were analyzed, revealing common features that contribute to their gender assignment.

Limitations and Future Work

  • Cross-Linguistic Extension: The authors acknowledge that their study is focused on French and suggest that a cross-linguistic extension would be valuable to explore how these methods generalize to other languages with grammatical gender.
  • Human-in-the-Loop Curation: The paper mentions the potential for human-in-the-loop curation to identify relevant new auxiliary attributes, suggesting a path for further refinement of the explanation system.
  • Consistency of Local Explanations: While local explanations can outperform global ones, reducing neighborhood size to improve interpretability can lead to some inconsistencies between explanations for nearby points in the feature embedding space. This is noted as a potential issue for applications where consistent explanations are crucial.

Summary updated

BibTeX
@inproceedings{sahai2021predicting,
  author = {Saumya Sahai and Dravyansh Sharma},
  title = {{Predicting and Explaining French Grammatical Gender}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure