Modeling Input Uncertainty in Neural Network Dependency Parsing

Rob van der Goot, Gertjan van NoordEMNLP 2018

Abstract

Neural dependency parsers often struggle with the "noisy" language characteristic of social media, such as typos, slang, and abbreviations. While modern parsers utilize character-level embeddings to mitigate these issues, this paper demonstrates that explicit lexical normalization remains a vital preprocessing step. We introduce a novel "Integrated" approach that models input uncertainty by considering multiple normalization candidates simultaneously through linear interpolation. Our results show that this method significantly boosts parsing accuracy on noisy data, nearly reaching the performance of gold-standard manual normalization.

Introduction

Dependency parsing—the task of identifying the grammatical structure of a sentence—has achieved high accuracy on standard, "clean" text like news articles. However, performance typically plummets when applied to social media data. Modern neural parsers attempt to address this using character-level embeddings, which allow models to recognize internal word structures (e.g., identifying that "goooood" relates to "good").

This research challenges the assumption that sub-word features make traditional lexical normalization obsolete. We investigate whether explicit normalization still provides value to neural models and propose a method to better handle the uncertainty inherent in choosing a single "correct" normalization candidate.

Methodology

We combined two state-of-the-art tools: MoNoise (a lexical normalization system) and UUParser 2.0 (a transition-based neural dependency parser). We evaluated four distinct input strategies:

  1. ORIG: Feeding the parser raw, noisy tokens.
  2. NORM: Using only the single highest-probability normalization candidate as a preprocessing step.
  3. INTEGRATED: Our novel approach. Instead of selecting a single winner, we take the top-N normalization candidates and their associated probabilities. We create a new input vector for the parser by calculating a linear interpolation (a weighted average) of the word embeddings for all top candidates. This allows the parser to account for the uncertainty of the normalization step.
  4. GOLD: Using manually corrected normalization to establish a theoretical upper bound.

Experimental Setup

To address the lack of noisy benchmarks with consistent dependency annotations, we created and released a new treebank consisting of 327 tweets (10,005 tokens). These tweets are annotated with both lexical normalization and Universal Dependencies (UD) 2.1 labels. We also utilized the English Web Treebank to ensure our model remains robust on canonical text.

Results and Key Insights

Our experiments yielded several critical insights into the relationship between normalization and neural parsing:

  • Normalization is Complementary: Character-level embeddings are not a silver bullet. Normalization provides a distinct signal that improves parsing even when character features and external word embeddings are already present.
  • The "Root" of the Problem: Normalization is most beneficial at the beginning of a sentence. In noisy text, parsers often fail to identify the main verb or subject (the "root"); normalization clarifies the structure enough for the parser to find its footing.
  • Hedging Bets Wins: The "Integrated" approach outperformed the single-best-candidate approach. By providing a weighted blend of possibilities, the parser can recover when the normalization tool ranks the correct word as a secondary choice.
LAS performance across different embedding combinations and normalization strategies, highlighting the additive benefit of normalization.

On the noisy Twitter test data, the performance improvements were marked. The "Integrated" approach nearly reached the theoretical maximum performance of gold-standard normalization.

Final parsing performance (UAS and LAS) on noisy Twitter data.

Conclusion and Future Work

Modeling input uncertainty is a highly effective strategy for parsing noisy text. Our method is robust; when applied to clean English, the normalization model correctly identifies that no changes are needed, ensuring performance does not degrade on standard text.

Future work will address current limitations, such as the 10,000-token dataset size and the current model's inability to handle complex normalization cases like word splitting (e.g., "wanna" to "want to") or merging. Additionally, we aim to expand this approach to morphologically rich languages where non-standard language presents unique structural challenges.

Summary updated

BibTeX
@inproceedings{goot2018modeling,
  author = {Rob van der Goot and Gertjan van Noord},
  title = {{Modeling Input Uncertainty in Neural Network Dependency Parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure