Generalizable Architecture for Robust Word Vectors Tested by Noisy Paraphrases

Valentin MalykhAIST (Supplement) 2017 · OpenReview

Abstract

Traditional word embedding models, such as word2vec, are highly sensitive to out-of-vocabulary (OOV) tokens caused by typos or morphological variations. This paper introduces a character-aware neural architecture that dynamically generates word embeddings using a Begin-Middle-End (BME) representation and LSTM layers. By focusing on the internal structure of words rather than fixed lookups, the model demonstrates significant robustness to synthetic noise. Evaluation across Russian, English, and Turkish paraphrase corpora shows that while standard models may excel on clean text, the proposed robust architecture maintains semantic performance even under heavy corruption, outperforming baselines as noise levels increase.

Introduction

Most word embedding frameworks rely on a fixed vocabulary, making them brittle in the face of real-world text noise. A single typo can transform a known word into an OOV token, stripping it of all semantic meaning. While sub-word models offer some improvement, they often fail to capture the specific patterns of noise common in mobile typing or the complex affixation of morphologically rich languages. This research proposes a language-independent architecture designed to be "indifferent" to such typos by generating embeddings on-the-fly from character-level information.

Methods: The BME-LSTM Architecture

The proposed model moves away from static embedding tables toward a dynamic generation pipeline. The core of this approach is the Begin-Middle-End (BME) representation, which captures the "skeleton" of a word:

  • Begin (B): A one-hot encoding of the first three letters.
  • End (E): A one-hot encoding of the last three letters.
  • Middle (M): A "bag-of-characters" (sum of one-hot vectors) for all letters in the word.

This structure is specifically designed to preserve the most informative parts of a word (prefixes and suffixes) while remaining resilient to internal character swaps or omissions. The BME input is processed through two fully-connected layers and a stack of three Long Short-Term Memory (LSTM) layers to produce the final fixed-size word vector.

The BME-LSTM architecture for generating character-aware word embeddings from character sequences.

The system is trained using a Continuous Bag-of-Words (CBOW) objective with negative sampling, learning to predict target word vectors based on their surrounding context.

Experimental Evaluation

The architecture was evaluated on paraphrase identification tasks across three typologically distinct languages: Russian (synthetic flexive), English (analytical), and Turkish (synthetic agglutinative). To simulate real-world conditions, researchers injected synthetic noise—randomly inserting or deleting characters—at levels ranging from 0% to 30%.

Key Results and Robustness

A primary finding of the study is the difference in degradation patterns between models. Standard word2vec models typically perform well on clean text but experience a "catastrophic failure" as noise is introduced. In contrast, the robust BME-LSTM architecture shows a much steadier, linear decrease in quality.

ROC AUC performance vs. noise level for the Russian corpus, illustrating the crossover point where the robust architecture begins to outperform standard baselines.

In the Russian corpus, the robust model began outperforming the standard baseline at a noise level of just 6%. While the standard model's performance became essentially random at 14% noise, the robust model remained effective up to the 30% testing limit. Similar trends were observed in English, where the robust architecture took the lead at approximately 14-16% noise.

Beyond robustness, the model is remarkably efficient. While the Google News word2vec model contains approximately 1 billion parameters, this architecture achieves its results with only 3 million parameters, making it suitable for resource-constrained environments.

Conclusion and Future Work

The BME-LSTM architecture provides a generalizable solution for generating robust word vectors that survive significant text corruption. Its performance is particularly notable in morphologically rich languages where word endings carry critical grammatical information. Future research will focus on narrowing the performance gap on clean text and further optimizing the architecture for complex agglutinative languages like Turkish, where the current BME setup may require additional refinement to capture intricate internal word structures.

Summary updated

BibTeX
@inproceedings{malykh2017generalizable,
  author = {Valentin Malykh},
  title = {{Generalizable Architecture for Robust Word Vectors Tested by Noisy Paraphrases}},
  booktitle = {AIST (Supplement)},
  year = {2017},
  url = {https://openreview.net/forum?id=wEVS3XVJgk},
}
Figure