TL;DR
This paper explores two distinct approaches to automatically tagging the grammatical role of words (Part-of-Speech tagging) in English-Assamese "code-mixed" text, common in social media. They found that building a dedicated system for code-mixed data, rather than relying on separate English and Assamese tools, is more effective and potentially leads to better overall language technology development, especially for low-resource languages.
What Problem Does This Paper Address?
Code-mixing, the practice of blending languages within a single sentence or conversation, is a pervasive phenomenon, especially in multilingual societies and online communication. Automatically processing this mixed-language text presents a significant challenge for natural language processing (NLP) tasks like Part-of-Speech (POS) tagging. Traditional NLP systems are built for "clean," monolingual data. Applying these systems directly to code-mixed text, or trying to combine them, often leads to poor performance. This paper tackles the problem of developing effective POS taggers for English-Assamese code-mixed text by comparing two main strategies: an "ensemble" approach using existing monolingual tools and a "novel multilingual" approach that builds a single system specifically for code-mixed data.
Methods
The authors investigate two primary approaches:
- Monolingual Ensemble Approach: This method leverages existing, pre-trained POS taggers for individual languages (English and Assamese). The process involves:
- Language Identification: Determining the language of each word.
- Script Identification: Identifying the script (e.g., Roman for Assamese, Latin for English).
- Transliteration: Converting Roman script Assamese to its native script (a crucial step as monolingual systems often expect native scripts).
- Language-Specific POS Tagging: Applying the appropriate monolingual POS tagger based on the identified language.
- Novel Multilingual Approach: This approach focuses on building a single POS tagger trained directly on the code-mixed dataset from scratch. This method bypasses the need for separate language identification and transliteration modules.
For their experiments, they used a Linear SVM classifier with a combination of word-level features (current word, surrounding words), tag-level features (previous tags), character-level features (prefixes/suffixes), and boolean features (e.g., presence of hyphens, numeric status).
Datasets
- Code-mixed Dataset: A new dataset was collected from four public Facebook pages containing English-Assamese code-mixed text. This dataset was annotated at the word level with both language identification and Part-of-Speech tags.
- Monolingual Assamese Dataset: Acquired from the Indian Languages Corpora Initiative (ILCI), this dataset was originally annotated with the Bureau of Indian Standards (BIS) tagset. The authors developed a mapper to convert BIS tags to the Universal Dependencies (UD) tagset for consistency.
- Monolingual English Dataset: Obtained from the CoNLL 2018 shared task, this dataset was already annotated using the Universal Dependencies tagset.
Key Insights and Ideas
- The "Clean Data" Fallacy: The paper challenges the assumption that monolingual, "clean" data is always the best foundation for NLP tools. For languages with limited resources, developing technologies directly on code-mixed data might be more efficient and ultimately yield better results.
- Code-Mixing as the Norm: The authors argue that code-mixed data is becoming the norm, especially in social media. Therefore, building technologies that can handle this directly is crucial for future applications.
- Ensemble Complexity vs. Novelty: The ensemble approach, while seemingly logical, introduces a complex pipeline with multiple dependencies (language ID, transliteration). Errors in any of these stages can propagate and significantly degrade overall performance.
- Transliteration Hurdles: The lack of a readily available Roman-to-Assamese transliteration system was a significant bottleneck for the ensemble approach, forcing them to use an approximation (Roman Bangla) which likely impacted performance.
- Potential for Superior Performance: The paper suggests that technologies trained on code-mixed data might perform "overall" better, even on monolingual texts, compared to systems trained solely on monolingual data.
Key Results
- Monolingual Performance: When tested on their own language data, monolingual taggers achieved high F1 scores (Assamese: 0.90, English: 0.88). The code-mixed tagger, trained on a smaller code-mixed dataset, performed slightly worse (0.84 F1).
- Code-mixed Tagger on Monolingual Data: A tagger trained on code-mixed data showed a performance drop when tested on monolingual Assamese (0.64 F1) and English (0.65 F1) datasets, as expected.
- Ensemble Approach Performance: The ensemble approach, using separate monolingual taggers, resulted in a significant performance drop when applied to code-mixed data (F1 of 0.50). This was attributed to errors in language identification and the imperfect transliteration system.
- Novel Multilingual Approach Superiority: The tagger trained directly on the code-mixed dataset outperformed the ensemble approach on code-mixed data (F1 of 0.84 vs. 0.50).
Limitations and Future Work
- Dataset Size: The authors acknowledge that the code-mixed dataset used for training was relatively small, which likely contributed to the lower performance of the code-mixed tagger when tested on monolingual data.
- Transliteration System: The reliance on an approximate transliteration system for Assamese was a significant limitation for the ensemble approach.
- Error Propagation: The paper highlights the issue of error propagation in pipelines, suggesting that a single, end-to-end system (like the novel multilingual approach) is less susceptible to this.
- Future Work: The authors plan to manually transliterate the test set to assess the performance of the ensemble approach with perfect transliteration. They also suggest that building new technologies directly on code-mixed data is a promising direction, especially for languages lacking extensive NLP resources.
Summary updated