Adapting lexical representation and OOV handling from written to spoken language with word embedding

Jeremie Tafforeau, Thierry Artieres, Benoit Favre, Frederic BechetInterspeech 2015

TL;DR

This paper tackles the challenge of adapting word embeddings, typically trained on vast written text corpora, to the nuances of spontaneous spoken language, which often has different vocabulary and usage patterns. They propose a novel method to refine these embeddings for a specific task and, crucially, to generalize this refinement to out-of-vocabulary (OOV) words not seen during task-specific training. This approach significantly improves Part-of-Speech (POS) tagging performance on spontaneous speech, especially when limited annotated data is available.

What Problem Does This Paper Address?

Word embeddings, like those from Word2Vec, have revolutionized NLP by representing words as dense vectors that capture semantic and syntactic relationships. A key assumption is that words with similar meanings or roles have similar embeddings, enabling models to generalize to unseen words. However, embeddings are typically trained on massive, clean, written text corpora (e.g., Wikipedia). This creates a mismatch when applying them to spontaneous speech, which is characterized by disfluencies, informal language, and different word usage (e.g., "like" as an adverb, frequent use of "yes"/"no").

When adapting these written embeddings to a specific spoken language task (like POS tagging), two main issues arise:

  1. Domain Mismatch: The learned embeddings might not accurately reflect the context and meaning of words in spontaneous speech.
  2. Out-of-Vocabulary (OOV) Words: Even after adaptation, a significant number of words in the spoken data might not be present in the original embedding vocabulary or the limited task-specific training data, leading to poor performance.

The paper aims to bridge this gap by developing a method that not only adapts existing embeddings to spoken language but also intelligently handles OOV words.

Methods

The authors propose a two-pronged approach to adapt word embeddings for spoken language POS tagging using a neural network:

  1. Embedding Refinement (ER): They integrate word embeddings as the input layer of a neural network for POS tagging. The initial embeddings are taken from a pre-trained model (like Word2Vec). During the training of the neural network on the target POS tagging task, these embeddings are treated as trainable parameters and are refined. This process allows the embeddings to become more task- and domain-specific. This is applied to words present in the task-specific training corpus.
  2. Artificial Refinement of OOV Embeddings (AER): This is the core novelty. For words that are out-of-vocabulary (OOV) in the task-specific training data (i.e., not seen during ER), they propose an "artificial refinement." The idea is that if a word is OOV, its embedding can be approximated by the refined embeddings of its closest neighbors in the original embedding space. Specifically, they calculate the shift applied to the embeddings of known words during ER (i.e., refined_embedding - original_embedding). For an OOV word, they compute a weighted sum of these shifts from its K nearest neighbors in the original embedding space, proportional to their cosine similarity. This artificially refines the OOV word's embedding, making it more aligned with the spoken language domain.

The neural network architecture itself is a standard feed-forward network with two hidden layers, taking a window of words and morphological features as input.

Datasets

  • Unlabelled Corpus (Cemb): French part of Wikipedia (357 million words) – used for pre-training the initial word embeddings.
  • In-domain Labelled Corpus (Ctask): French RATP-DECODA corpus (546K words) – a collection of phone conversations from the Paris public transport call center, used for training and evaluating the POS tagger.

The authors created different training set sizes from Ctask (D10, D50, D100, Dall) to evaluate the impact of limited annotated data. They also categorize test words into W1 (known, in Ctask), W2 (known, in Cemb but not Ctask), W3 (OOV, in Ctask but not Cemb), and W4 (OOV, not in Cemb or Ctask).

Key Insights and Ideas

  • Spoken Language is Different: The paper strongly emphasizes that standard written embeddings are a poor fit for spontaneous speech, leading to performance degradation.
  • Generalizing Refinement is Crucial: The most significant insight is that simply refining embeddings for words seen in the task data isn't enough. The ability to intelligently infer and adapt embeddings for unseen (OOV) words is key to achieving robust performance.
  • Neighbor-Based OOV Adaptation: The idea of using the "wisdom of the crowd" (nearest neighbors) in the embedding space to "guess" the refined embedding for an OOV word is a clever and effective heuristic. It leverages the existing semantic structure to adapt to a new domain.
  • Task-Specific Embeddings are Better: The results show that even a small amount of task-specific adaptation (ER) significantly improves performance, and further extending this to OOV words (AER) provides consistent gains.

Key Results

The proposed method (NN+ER+AER) consistently outperforms the state-of-the-art Conditional Random Field (CRF) baseline, especially when training data is scarce (D10 and D50).

  • Error Rate Reduction: For the D10 split (very limited data), the error rate for W1 and W2 words was reduced from 32.6% (CRF) to 21% (NN+ER+AER).
  • Consistent Gains: Even with the full training corpus (Dall), the NN+ER+AER system achieved a lower error rate (8.7%) compared to the CRF baseline (10.9%).
  • Individual Component Gains: Table 4 shows that the ER component alone provides significant improvements, and the AER component adds a constant gain, validating both parts of their strategy.

Limitations and Future Work

  • OOV Categories W3 and W4: The authors acknowledge that their approach primarily addresses OOV words that have some representation in the original embedding space (W2). They did not present results for W3 and W4 (words completely absent from the embedding corpus), noting that this is orthogonal to their approach but might explain the CRF's slightly better overall performance in some cases.
  • Future Work:
    • Global Optimization: Incorporating tag sequence global optimization using Recurrent Neural Network (RNN) architectures and Long-Short Term Memory (LSTM) neurons, rather than treating each word independently.
  • Broader Tasks: Investigating the relevance of their approach on more complex NLP tasks like syntactic parsing and semantic role labeling.
  • Handling Truly Unknown Words: Developing specific processes to account for words completely absent from the embedding vocabulary.

Summary updated

BibTeX
@inproceedings{tafforeau2015adapting,
  author = {Jeremie Tafforeau and Thierry Artieres and Benoit Favre and Frederic Bechet},
  title = {{Adapting lexical representation and OOV handling from written to spoken language with word embedding}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure