TL;DR
This paper introduces IndoCollex, a novel benchmark dataset for normalizing Indonesian colloquial words, addressing the significant gap between formal and informal language in NLP. They evaluate various character-level models, finding that while Transformers show promise, colloquial transformation, especially from formal to informal, remains a challenging task due to inherent ambiguity.
What Problem Does This Paper Address?
Indonesian, a widely spoken language, is heavily characterized by colloquialisms, particularly in informal online text. This linguistic variation poses a significant challenge for Natural Language Processing (NLP) systems, which are often trained on formal language. Existing approaches to Indonesian word normalization have largely relied on static dictionaries, limiting their ability to handle unseen words, or have focused on specific tasks like sentiment analysis without addressing the core normalization problem. The authors identify a critical need for a standardized, realistic testbed to evaluate and improve models for Indonesian colloquial word normalization, especially for morphological transformations that deviate from standard forms.
Methods
The paper proposes a new dataset, IndoCollex, which consists of Indonesian colloquial words annotated with their standard forms and word formation types. They then evaluate several character-level sequence-to-sequence (Seq2Seq) models for morphological word normalization. These include:
- BiLSTM+Dict: A Bi-directional LSTM model with dictionary constraints, considered a strong baseline for character-level transduction.
- Transformer: The Transformer architecture, known for its success in various sequence-to-sequence tasks, is evaluated for its ability to handle morphological transformations.
- Rule-Based Transformation: A manually defined system based on observed patterns in Indonesian colloquialisms.
Experiments were conducted in both formal-to-informal (F→I) and informal-to-formal (I→F) directions. They also explored the impact of providing word formation tags as input to the models and investigated the use of back-translation for data augmentation.
Datasets
- IndoCollex: The primary contribution is this newly created benchmark dataset. It comprises parallel formal and informal Indonesian word pairs, expertly annotated with word formation types. The data is sampled from Twitter across different regions and includes 9 types of word formation mechanisms (e.g., disemvoweling, shortening, phonetic alteration, compounding).
- Indonesian Wikipedia: Used to construct a formal lexicon for OOV (Out-Of-Vocabulary) rate calculations.
- English Tweets and Wikipedia: Used for comparative analysis of OOV rates and perplexity with Indonesian datasets.
Key Insights and Ideas
- Indonesian Colloquialism is Pervasive: The OOV rate for colloquial Indonesian text is significantly higher (double) than for informal English, and its perplexity is also considerably higher, indicating a more prominent issue with non-standard word formation.
- F→I Transformation is More Ambiguous: Transforming formal words into informal ones is inherently more challenging than the reverse. This is attributed to the higher degree of ambiguity, where a single formal word can have multiple valid informal transformations.
- Word Formation Type Matters: Providing explicit word formation tags as input significantly improves the performance of both BiLSTM and Transformer models.
- Transformer Outperforms: The Transformer architecture generally outperforms BiLSTM models, especially when augmented with back-translation and word formation tags.
- Rule-Based Models Have Niche Strengths: Surprisingly, the rule-based approach achieved competitive top-1 accuracy, though it struggled with top-10 accuracy due to its inability to generate multiple candidates.
Key Results
- The Transformer model with both back-translation and word formation tags achieved the best test performance in both F→I and I→F directions.
- Top-1 accuracy for the best Transformer model reached 37.4% for Informal to Formal and 36.6% for Formal to Informal.
- The Transformer model's accuracy in predicting word formation types was 81.4% for I→F and 65.0% for F→I, highlighting the difficulty of the F→I direction.
- Indonesian tweets showed approximately twice the OOV rate (14.6%) compared to English tweets (6.4%).
Limitations and Future Work
- Data Bias: The current IndoCollex dataset is largely Jakarta-centric, despite efforts to collect data from various provinces. Further scraping and verification are needed to ensure broader linguistic coverage.
- Contextual Information: The current models primarily rely on word-level transformations. Future work aims to incorporate contextual information (sentences, images, memes) to improve word transformation, potentially by learning mappings between formal and informal words based on sentence context similarities.
- Downstream Applications: The authors suggest using text normalization to augment data for informal text translation and for training downstream applications like framing identification on informal social media text.
- Ambiguity in Transformations: Certain word formation mechanisms, like acronyms, can be ambiguous, making it challenging for models to consistently reconstruct the original phrase without explicit tags or more context.
- Affixation Challenges: The model sometimes struggles with affixation, mistaking common prefixes for root words, suggesting a need for more training data to capture various affixation patterns.
Summary updated