OTEANN: Estimating the Transparency of Orthographies with an Artificial Neural Network

Xavier MarjouNAACL 2021

TL;DR

This paper introduces OTEANN, a novel Artificial Neural Network (ANN) model designed to quantify the "transparency" of orthographies – how consistently written words map to their pronunciation. By training a single model on multiple languages, they provide a unified framework for ranking orthographies by their ease of reading and writing, offering valuable insights for language learning and NLP.

What Problem Does This Paper Address?

Spoken languages are transcribed using writing systems, ideally with a clear, one-to-one mapping between sounds (phonemes) and letters (graphemes). This property, known as orthographic transparency, makes learning to read and write significantly easier. However, many real-world orthographies, like English and French, have evolved to incorporate complex rules and exceptions, leading to ambiguity and making them "opaque." While linguists have long discussed orthographic transparency, there's a lack of computational methods within Natural Language Processing (NLP) to objectively measure and compare this transparency across different languages. This gap hinders our ability to understand the challenges faced by language learners and to develop more effective NLP tools for diverse linguistic systems.

Methods

The core of this work is the OTEANN (Orthographic Transparency Estimation with an ANN) model. OTEANN is built upon a Sequence-to-Sequence (seq2seq) architecture, specifically a minimalist implementation of the GPT (Generative Pre-Training) model. The authors frame the tasks of writing (pronunciation to spelling) and reading (spelling to pronunciation) as translation problems.

Instead of training separate models for each language and task, OTEANN employs a single, multi-orthography ANN instance. This model is trained simultaneously on data from all studied orthographies. The input to the ANN is a concatenation of the orthography, the task (write/read), and the input word (either phonemes or graphemes). The ANN then predicts the corresponding output (graphemes or phonemes). This approach allows for efficient learning and direct comparison of transparency scores across languages.

The performance metric used is the percentage of correct predictions for each orthography-task pair. A score of 0% indicates a completely opaque orthography, while 100% signifies perfect transparency.

Datasets

The study utilizes a custom-built multi-orthography dataset. This dataset was constructed by combining data from various sources, primarily Wikimedia dictionaries (Wiktionary dumps). For each orthography, they created two sub-datasets: one for the "write" task (phoneme sequence to grapheme sequence) and one for the "read" task (grapheme sequence to phoneme sequence).

To establish baselines, they created two artificial orthographies:

  • Entirely Transparent ('ent'): Mimics Esperanto, where phonemes and graphemes have a perfect one-to-one mapping.
  • Entirely Opaque ('eno'): Assigns random graphemes to phonemes, ensuring no correlation.

The studied orthographies include: Arabic, Breton, German, English, Esperanto, Spanish, Finnish, French, Italian, Korean, Dutch, Portuguese, Russian, Serbo-Croatian, Turkish, and Chinese. They also included a proposed phonemically transparent orthography for French ('fro').

For training and testing, they randomly selected 11,000 samples from each of the 17 sub-datasets. Each sample from a sub-dataset generated two instances in the multi-orthography dataset (one for writing, one for reading). The dataset was then split into training (10,000 samples per orthography-task pair) and testing (1,000 samples per orthography-task pair) sets.

Key Insights and Ideas

  • Unified ANN for Multi-Orthography Analysis: The most significant innovation is using a single ANN to learn and compare transparency across many orthographies simultaneously. This avoids the need for language-specific models and provides a consistent evaluation framework.
  • Quantifying Transparency: OTEANN provides a concrete, data-driven way to rank orthographies by their transparency, moving beyond qualitative linguistic descriptions.
  • Learner Error Insights: The model's predictions reveal typical mistakes made by learners who might over-rely on purely phonemic rules, offering insights into common learning difficulties.
  • Asymmetrical Transparency: The study highlights that writing and reading tasks are not always symmetrical in their transparency. An orthography might be easier to read than to write, or vice-versa.
  • Cost of Opacity: The results suggest that learning opaque orthographies requires significantly more training data and computational resources for the ANN to achieve comparable performance to transparent ones.

Key Results

  • Baseline Validation: The 'ent' orthography achieved near-perfect scores (above 99.6%), validating the model's ability to capture perfect transparency. The 'eno' orthography scored 0%, confirming its complete opacity.
  • High Transparency: Orthographies like Esperanto (above 99.3%), Arabic, Finnish, Korean, Serbo-Croatian, and Turkish (above 80%) were confirmed as highly transparent.
  • Moderate Transparency: Breton, German, Italian, Portuguese, and Spanish showed fair transparency (above 65%).
  • Low Transparency (Opaque): English (writing: 36%, reading: 31%) and French (writing: 28%, reading: 80%) were found to be highly opaque, especially for the writing task. Chinese also showed a low writing score (20%).
  • Comparison with Prior Work: The reading task results for Dutch, English, and French aligned with previous studies (van den Bosch et al., 1994), reinforcing the model's validity.
  • Learning Curves: Figure 9 demonstrates that increasing training samples significantly improves scores, especially for opaque orthographies, highlighting the "cost" of learning them.

Limitations and Future Work

  • IPA Symbol Consistency: The study acknowledges potential inconsistencies in IPA symbol usage across different dictionaries, which could affect accuracy.
  • Ambiguity and Context: The ANN's inability to consider broader context (beyond single words) limits its ability to resolve homophones with different spellings, a common issue in opaque orthographies.
  • Scoring Formula: The authors suggest that the scoring formula might need revision to better account for certain types of errors, particularly those involving accents or subtle vowel distinctions.
  • Dataset Quality: The Wiktionary data, while extensive, may have quality variations that could impact results.
  • Future Directions:
    • Investigate alternative accuracy metrics beyond simple exact matching.
  • Explore the impact of using a single ANN instance versus separate models per orthography.
  • Analyze the phonemes used for each orthography more closely to understand their influence on scores.
  • Use OTEANN to evaluate proposed improvements to opaque orthographies or to detect errors in transparent ones.
  • Investigate if the ANN's learned representations mimic how human beginners learn languages.

Summary updated

BibTeX
@inproceedings{marjou2021oteann,
  author = {Xavier Marjou},
  title = {{OTEANN: Estimating the Transparency of Orthographies with an Artificial Neural Network}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure