Abstract
Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, Wladimir Sidorenko. Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021).
2021.
Introduction
Natural Language Processing (NLP) tools are typically trained on "canonical" text, such as news articles or Wikipedia. However, user-generated content on social media is a "wild west" of linguistic variation, featuring slang, abbreviations, typos, and code-switching. Standard parsers often fail on sentences like "u hve to let ppl decide" because they do not recognize "u" as "you" or "ppl" as "people."
Lexical normalization is the task of translating this noisy text into its standardized form. While studied for over a decade, research in this area has been fragmented by inconsistent datasets and metrics. MultiLexNorm addresses this by providing the largest unified benchmark for multilingual lexical normalization, covering 12 language variants.
Methodology and Datasets
The shared task involved nine teams submitting 18 systems. The benchmark includes 10 standard languages (Danish, German, English, Spanish, Croatian, Italian, Dutch, Slovenian, Serbian, and Turkish) and two code-switched pairs (Indonesian-English and Turkish-German).
Organizers provided three baselines:
- LAI (Leave-As-Is): No changes made.
- MFR (Most-Frequent-Replacement): A dictionary-based approach.
- MoNoise: The previous state-of-the-art using a random forest classifier.
The winning system, ÚFAL-2, utilized ByT5, a byte-level version of the T5 transformer. By operating on raw bytes rather than a fixed vocabulary, ByT5 is exceptionally robust to the out-of-vocabulary issues prevalent in noisy text. The team also employed a two-step fine-tuning process using synthetic noise generated from Wikipedia edits.
Key Results
The intrinsic performance of the systems was measured using the Error Reduction Rate (ERR). The winning ÚFAL-2 system achieved a macro-averaged ERR of 67.3%, a substantial improvement over the MoNoise baseline of 49.0%.
Extrinsic evaluation also demonstrated that normalization provides significant boosts for downstream tasks. For instance, the winning system improved labeled attachment scores (a-LAS) for dependency parsing by up to 1.72 points and POS tagging accuracy by up to 1.54 points. Performance gains were most pronounced in languages with high inherent noise, such as Dutch (28.29% change rate) and Turkish (37.02% change rate).
Insights and Future Directions
The success of byte-level transformers suggests that traditional tokenization is often a bottleneck for noisy text. By processing raw bytes, models can learn phonetic and orthographic patterns without dictionary constraints. Additionally, the use of synthetic data proved vital for pre-training models on the nature of linguistic noise.
Despite these advancements, normalization is not yet a "solved" problem. A performance gap remains between normalized social media text and native canonical text. Future research should focus on automating tokenization—which was provided as "gold" in this task—and expanding the benchmark to include more non-Indo-European languages and complex transformations like word insertions or deletions.
Summary updated