MultiLexNorm: A Shared Task on Multilingual Lexical Normalization

Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Müller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, Wladimir Sidorenko2021 · DOI

Abstract

Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, Wladimir Sidorenko. Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021).

2021.

Introduction

Natural Language Processing (NLP) tools are typically trained on "canonical" text, such as news articles or Wikipedia. However, user-generated content on social media is a "wild west" of linguistic variation, featuring slang, abbreviations, typos, and code-switching. Standard parsers often fail on sentences like "u hve to let ppl decide" because they do not recognize "u" as "you" or "ppl" as "people."

Lexical normalization is the task of translating this noisy text into its standardized form. While studied for over a decade, research in this area has been fragmented by inconsistent datasets and metrics. MultiLexNorm addresses this by providing the largest unified benchmark for multilingual lexical normalization, covering 12 language variants.

Examples of lexical normalization across various languages, showing the mapping from noisy to standard forms.

Methodology and Datasets

The shared task involved nine teams submitting 18 systems. The benchmark includes 10 standard languages (Danish, German, English, Spanish, Croatian, Italian, Dutch, Slovenian, Serbian, and Turkish) and two code-switched pairs (Indonesian-English and Turkish-German).

Organizers provided three baselines:

  1. LAI (Leave-As-Is): No changes made.
  2. MFR (Most-Frequent-Replacement): A dictionary-based approach.
  3. MoNoise: The previous state-of-the-art using a random forest classifier.

The winning system, ÚFAL-2, utilized ByT5, a byte-level version of the T5 transformer. By operating on raw bytes rather than a fixed vocabulary, ByT5 is exceptionally robust to the out-of-vocabulary issues prevalent in noisy text. The team also employed a two-step fine-tuning process using synthetic noise generated from Wikipedia edits.

Key Results

The intrinsic performance of the systems was measured using the Error Reduction Rate (ERR). The winning ÚFAL-2 system achieved a macro-averaged ERR of 67.3%, a substantial improvement over the MoNoise baseline of 49.0%.

Intrinsic evaluation results (Error Reduction Rate) for the shared task. The table shows performance across all 12 language variants for the participating teams.

Extrinsic evaluation also demonstrated that normalization provides significant boosts for downstream tasks. For instance, the winning system improved labeled attachment scores (a-LAS) for dependency parsing by up to 1.72 points and POS tagging accuracy by up to 1.54 points. Performance gains were most pronounced in languages with high inherent noise, such as Dutch (28.29% change rate) and Turkish (37.02% change rate).

Insights and Future Directions

The success of byte-level transformers suggests that traditional tokenization is often a bottleneck for noisy text. By processing raw bytes, models can learn phonetic and orthographic patterns without dictionary constraints. Additionally, the use of synthetic data proved vital for pre-training models on the nature of linguistic noise.

Despite these advancements, normalization is not yet a "solved" problem. A performance gap remains between normalized social media text and native canonical text. Future research should focus on automating tokenization—which was provided as "gold" in this task—and expanding the benchmark to include more non-Indo-European languages and complex transformations like word insertions or deletions.

Summary updated

BibTeX
@misc{goot2021multilexnorm,
  author = {Rob van der Goot and Alan Ramponi and Arkaitz Zubiaga and Barbara Plank and Benjamin Müller and Iñaki San Vicente Roncal and Nikola Ljubešić and Özlem Çetinoğlu and Rahmad Mahendra and Talha Çolakoğlu and Timothy Baldwin and Tommaso Caselli and Wladimir Sidorenko},
  title = {{MultiLexNorm: A Shared Task on Multilingual Lexical Normalization}},
  year = {2021},
  doi = {10.18653/v1/2021.wnut-1.55},
}
Figure