Abstract
Modern natural language processing (NLP) faces a significant challenge: the prevalence of noisy text in real-world data. Traditional approaches rely on spelling correction, which is often imperfect. We propose RoVe (Robust Vectors), a word embedding model designed to be inherently resilient to noise. RoVe outperforms commonly used models like fastText and word2vec across various tasks. Furthermore, we investigate the noise robustness of current models in text classification, named entity recognition, and aspect extraction, proposing extensions that significantly improve performance in noisy environments.
Introduction
NLP models are typically trained on "clean" datasets like Wikipedia, but real-world inputs—especially from mobile devices and social media—are riddled with typos and misspellings. These create Out-of-Vocabulary (OOV) issues where models fail to recognize words they haven't seen in an exact form. Rather than attempting to "fix" the text via error-prone spell-checkers, this research introduces a model architecture that is robust by design.
Methodology: The RoVe Architecture
RoVe differs from standard embeddings by combining character-level morphological awareness with context-dependent encoding.
1. Open Vocabulary via BME Design
RoVe encodes words using a BME (Begin, Middle, End) representation. It treats the "Middle" of a word as a "bag-of-characters," making the model invariant to internal character swaps (e.g., recognizing "probelm" as "problem"). By encoding the "Begin" and "End" separately, the model maintains morphological awareness, which is crucial for synthetic languages like Russian or Turkish.
2. Context Dependency
Unlike static embeddings like word2vec, RoVe uses a bi-directional encoder (such as a biSRU) to analyze surrounding words. This allows for "implicit lemmatization," where the model uses context to determine the likely base concept of a misspelled word.
Experimental Setup
The study evaluates robustness across three linguistically diverse languages: English (analytical), Russian (synthetic fusional), and Turkish (synthetic agglutinative). Tasks included paraphrase detection, sentiment analysis, Natural Language Inference (SNLI), Named Entity Recognition (NER), and Aspect Extraction.
Key Results
RoVe consistently demonstrates superior performance as noise levels increase. While models like fastText may show a slight advantage on perfectly clean English data, RoVe becomes the superior choice once noise levels exceed approximately 7.5%.
In the Aspect Extraction task, the RoVe-extended version of the ABAE model showed significant improvements in $F_1$ scores across all noise levels compared to the baseline. This suggests that the BME approach is a viable, language-agnostic strategy for global NLP applications.
Conclusion and Future Work
RoVe provides a robust alternative to traditional embeddings by mimicking human cognitive resilience to typos. While highly effective for prefix- and suffix-heavy languages, future research will investigate its application to languages with infix morphology (like Arabic) and its utility in Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR) pipelines.
Summary updated