Robust to Noise Models in Natural Language Processing Tasks

Valentin MalykhACL 2019 · OpenReview

Abstract

Modern natural language processing (NLP) faces a significant challenge: the prevalence of noisy text in real-world data. Traditional approaches rely on spelling correction, which is often imperfect. We propose RoVe (Robust Vectors), a word embedding model designed to be inherently resilient to noise. RoVe outperforms commonly used models like fastText and word2vec across various tasks. Furthermore, we investigate the noise robustness of current models in text classification, named entity recognition, and aspect extraction, proposing extensions that significantly improve performance in noisy environments.

Introduction

NLP models are typically trained on "clean" datasets like Wikipedia, but real-world inputs—especially from mobile devices and social media—are riddled with typos and misspellings. These create Out-of-Vocabulary (OOV) issues where models fail to recognize words they haven't seen in an exact form. Rather than attempting to "fix" the text via error-prone spell-checkers, this research introduces a model architecture that is robust by design.

Methodology: The RoVe Architecture

RoVe differs from standard embeddings by combining character-level morphological awareness with context-dependent encoding.

The RoVe architecture combining BME word representation with context-dependent encoding.

1. Open Vocabulary via BME Design

RoVe encodes words using a BME (Begin, Middle, End) representation. It treats the "Middle" of a word as a "bag-of-characters," making the model invariant to internal character swaps (e.g., recognizing "probelm" as "problem"). By encoding the "Begin" and "End" separately, the model maintains morphological awareness, which is crucial for synthetic languages like Russian or Turkish.

2. Context Dependency

Unlike static embeddings like word2vec, RoVe uses a bi-directional encoder (such as a biSRU) to analyze surrounding words. This allows for "implicit lemmatization," where the model uses context to determine the likely base concept of a misspelled word.

Experimental Setup

The study evaluates robustness across three linguistically diverse languages: English (analytical), Russian (synthetic fusional), and Turkish (synthetic agglutinative). Tasks included paraphrase detection, sentiment analysis, Natural Language Inference (SNLI), Named Entity Recognition (NER), and Aspect Extraction.

Key Results

RoVe consistently demonstrates superior performance as noise levels increase. While models like fastText may show a slight advantage on perfectly clean English data, RoVe becomes the superior choice once noise levels exceed approximately 7.5%.

Performance comparison (F1 score) across varying noise levels showing RoVe's robustness.

In the Aspect Extraction task, the RoVe-extended version of the ABAE model showed significant improvements in $F_1$ scores across all noise levels compared to the baseline. This suggests that the BME approach is a viable, language-agnostic strategy for global NLP applications.

Conclusion and Future Work

RoVe provides a robust alternative to traditional embeddings by mimicking human cognitive resilience to typos. While highly effective for prefix- and suffix-heavy languages, future research will investigate its application to languages with infix morphology (like Arabic) and its utility in Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR) pipelines.

Summary updated

Abstract

There are a lot of noise texts surrounding a person in modern life. The traditional approach is to use spelling correction, yet the existing solutions are far from perfect. We propose robust to noise word embeddings model, which outperforms existing commonly used models, like fasttext and word2vec in different tasks. In addition, we investigate the noise robustness of current models in different natural language processing tasks. We propose extensions for modern models in three downstream tasks, i.e. text classification, named entity recognition and aspect extraction, which shows improvement in noise robustness over existing solutions.
BibTeX
@inproceedings{malykh2019robust,
  author = {Valentin Malykh},
  title = {{Robust to Noise Models in Natural Language Processing Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=6j0ci0jlVf},
}
Figure