Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers (BORT)

Robert C. Gale, Alexandra C. Salem, Gerasimos Fergadiotis, Steven BedrickACL 2023

TL;DR

This paper introduces BORT, a novel language model that can process both standard text and the International Phonetic Alphabet (IPA), a system for transcribing speech sounds. By extending existing language models with self-supervised tasks that teach them to translate between IPA and orthography, BORT significantly improves performance on clinical tasks requiring analysis of disordered speech, like that of individuals with aphasia.

What Problem Does This Paper Address?

Modern large language models (LLMs) have achieved remarkable success in understanding and generating text. However, their capabilities are largely confined to orthographic (written) representations of language. This is a significant limitation for applications that require understanding the nuances of spoken language, especially in clinical settings. For instance, speech-language pathologists often need to analyze both the semantic meaning and the phonetic realization of speech, particularly when dealing with speech errors. Current LLMs struggle with this because:

  1. Orthographic Bias: The vast majority of training data for LLMs is text-based, favoring writing systems like the English alphabet, which are not always a direct or precise representation of speech sounds.
  2. Limited IPA Support: LLMs have poor or non-existent support for the International Phonetic Alphabet (IPA), a standardized system crucial for accurately representing speech sounds.
  3. Tokenization Limitations: LLMs typically tokenize text at the word or sub-word level. This means they don't inherently capture phonetic information, and pre-training tasks like masked language modeling (MLM) gain little from phonetic representations.

This gap makes it difficult to apply LLMs to tasks like analyzing code-switched language, processing noisy speech recognition output, or, crucially, understanding speech disorders like aphasia, where both phonology and semantics are vital.

Methods

The core contribution of this paper is the development of BORT (Beyond Orthographically Restricted Transformers). BORT is built upon the BART (Bidirectional and Auto-Regressive Transformer) architecture, a powerful sequence-to-sequence model. The key innovation lies in extending BART's pre-training with novel self-supervised tasks designed to handle mixed orthographic and IPA inputs.

The authors implemented several pre-training configurations for BORT, primarily focusing on an IPA-to-orthography translation task. This involved:

  • Pronunciation Transform: Replacing words in Wikipedia articles with their IPA transcriptions. The model is then trained to restore the original orthography. To handle the tokenizer's tendency to merge IPA characters, bullet symbols were strategically inserted to enforce separation.
  • Spelling Transform: Inserting bullet symbols before each letter of a word to force the tokenizer to treat each letter individually. This was used to augment the model's understanding of orthographic representation.
  • Noise Transform: A denoising task, similar to BART's original pre-training, where tokens are randomly inserted, deleted, or replaced within words.

These transforms were applied in various combinations to different BORT models, with some configurations also incorporating noise. The goal was to create a model that could effectively process and understand language represented by a mix of English text and IPA.

Datasets

  • Wikipedia: Used as the primary source for orthographic text during pre-training. The 20220301.en version was utilized.
  • CMUDict (Carnegie Mellon University Pronouncing Dictionary): Version 0.7b was the foundation for the pronunciation dictionary. ARPABET entries were converted to IPA, with stress symbols removed. This dictionary was a key bottleneck, limiting the number of words that could be directly transformed into IPA.
  • AphasiaBank: This dataset, a widely-used repository of transcribed speech from individuals with aphasia (PWA), was used for fine-tuning and evaluating BORT on clinical tasks. It contains transcripts with both orthographic and IPA representations of speech, along with annotations for target words and error types (phonological, semantic, etc.).

Key Insights and Ideas

  • IPA is Crucial for Clinical NLP: The paper strongly emphasizes that for many clinical applications, especially in speech-language pathology, both phonological and semantic information are indispensable. LLMs that only handle orthography are inherently limited in these domains.
  • Self-Supervised Learning for Phonology: The authors demonstrate that self-supervised learning can be effectively adapted to teach LLMs about phonetics and phonology, not just syntax and semantics. The IPA-to-orthography translation task is a novel way to achieve this.
  • Spelling and Noise Help: Pre-training with spelling and noise transforms, in addition to pronunciation, proved beneficial. This suggests that a more robust representation, capable of handling variations and errors, leads to better downstream performance.
  • Context Matters for Evaluation: The paper highlights that strict hold-out rules during pre-training (to prevent memorization) can lead to lower CER scores than expected, but this doesn't necessarily reflect the model's utility in a broader pipeline.

Key Results

The paper presents results from both pre-training and fine-tuning stages.

Pre-training: * The best performing pre-trained model was BORT-PR-SP-NOISY, which was trained with pronunciation, spelling, and noise transforms. It achieved a Character Error Rate (CER) of 15.1% and an accuracy of 64.5% on the IPA-to-orthography restoration task. * Models incorporating noise generally outperformed those without it.

Fine-tuning (Clinical Task - Aphasia): * For the "easy" task (predicting the correct pronunciation for a given word), BORT-PR-SP achieved the highest accuracy of 90.1% and a CER of 5.7%. This represents a relative improvement of 24% in accuracy and 75% in CER compared to the baseline BART model. * For the "hard" task (predicting the intended word from a paraphasic production), the best performing model was BORT-PR-SP-NOISY, achieving 46.7% top 1 accuracy and 42.0% CER. This is a relative improvement of 29% in accuracy and 31% in CER over the baseline. * Crucially, all BORT models significantly outperformed the baseline BART model on both "easy" and "hard" tasks, demonstrating the effectiveness of the mixed orthographic/phonemic approach.

Limitations and Future Work

  • IPA Dictionary Size: The limited size of the pronunciation dictionary (98K words) was a bottleneck during pre-training, potentially leading to overfitting.
  • Character-Level Models: The authors suggest that character-level models (like CANINE) might offer advantages in handling the explicit demarcation of phonemic and orthographic regions, potentially overcoming some limitations of BORT.
  • More Realistic Noise: Future work could involve more sophisticated noise transforms, such as replacing phonemes with statistically or linguistically similar ones, and generating whole-word replacements with semantically related words to better mimic real-world speech errors.
  • Fine-grained Phonetics: The current models use a basic inventory of English phonemes. A more detailed phonetic analysis would require a more comprehensive pronunciation dictionary.
  • Non-English Languages: While the techniques are expected to be applicable to other languages, the current work is limited to English resources.
  • ASR Integration: The current clinical application assumes transcribed input. Integrating BORT with robust automatic speech recognition (ASR) systems capable of handling disordered speech is a significant area for future research.
  • Training Schedule: The pre-training did not always complete a full epoch through the Wikipedia data. Adjusting the training schedule (e.g., learning rate ramp-up) and task configurations could improve efficiency.

Summary updated

BibTeX
@inproceedings{gale2023mixed,
  author = {Robert C. Gale and Alexandra C. Salem and Gerasimos Fergadiotis and Steven Bedrick},
  title = {{Mixed Orthographic/Phonemic Language Modeling: Beyond Orthographically Restricted Transformers (BORT)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure