Creation of Corpus and Analysis in Code-Mixed Kannada-English Social Media Data for POS Tagging

Appidi Abhinav Reddy, Vamshi Krishna Srirangam, Darsi Suhas, Manish ShrivastavaICON 2020 · OpenReview

Abstract

Part-of-Speech (POS) is one of the essential tasks for many Natural Language Processing (NLP) applications. There has been a significant amount of work done in POS tagging for resource-rich languages. POS tagging is an essential phase of text analysis in understanding the semantics and context of language. These tags are useful for higher-level tasks such as building parse trees, which can be used for Named Entity Recognition, Coreference resolution, Sentiment Analysis, and Question Answering. There has been work done on code-mixed social media corpus but not on POS tagging of Kannada-English code-mixed data. Here, we present Kannada-English code-mixed social media corpus annotated with corresponding POS tags. We also experimented with machine learning classification models CRF, Bi-LSTM, and Bi-LSTM-CRF models on our corpus.

Introduction

In multilingual societies like India, social media users frequently engage in code-mixing—the practice of blending two or more languages, such as Kannada and English, within a single sentence or even a single word. Standard Natural Language Processing (NLP) tools are typically trained on monolingual data and often fail when faced with the informal, chaotic nature of social media text. For a language like Kannada, which is relatively low-resource in the digital space, the lack of annotated data makes it difficult to build reliable tools for downstream tasks like sentiment analysis or named entity recognition. This research addresses this gap by creating a gold-standard dataset and evaluating baseline models for POS tagging in code-mixed Kannada-English data.

Dataset Creation and Annotation

The researchers introduced a novel dataset consisting of 6,468 code-mixed tweets, filtered from a larger pool of over 318,000 tweets scraped over six years. The final corpus contains 156,761 tokens.

Annotation was performed manually by two proficient speakers using the Universal POS tagset, which includes 17 tags such as NOUN, VERB, and ADJ. To ensure the reliability of the dataset, the authors measured inter-annotator agreement using Cohen’s Kappa. As shown in the table below, most tags achieved a score above 0.80, indicating high quality and consistency in the annotations.

Inter-annotator agreement (Cohen's Kappa) and token distribution per tag.

Methodology

The study compared three distinct architectures for sequence labeling:

  1. Conditional Random Fields (CRF): A traditional discriminative model effective at capturing contextual features.
  2. Bi-LSTM: A deep learning model that processes text in both directions to capture long-range dependencies.
  3. Bi-LSTM-CRF: A hybrid model combining Bi-LSTM feature learning with CRF sequence optimization.

Key features engineered for these models included character and word N-grams (to handle misspellings and hybrid "Kanglish" words), contextual features, social media-specific tokens (hashtags, mentions), and capitalization patterns.

Results and Analysis

The models performed well despite the inherent complexity of code-mixed text. The Bi-LSTM-CRF model achieved a peak weighted average F1-score of 0.81, while the CRF and Bi-LSTM models followed closely with scores around 0.80.

A critical part of the analysis involved an ablation study to determine the impact of various features on the CRF model. The results demonstrated that Character N-Grams are the most vital feature for this task. Removing them caused the F1-score to drop drastically from 0.80 to 0.45. This highlights the importance of sub-word information in processing code-mixed and morphologically rich languages.

Ablation study showing the impact of removing specific features on model performance.

Linguistic Challenges

The study highlighted several challenges unique to code-mixed Kannada-English:

  • Morphological Mixing: Words like cinemagalu combine an English root ("cinema") with a Kannada plural suffix ("-galu").
  • Ambiguity: Words like "Bali" can refer to a location in English or mean "Near" in Kannada, requiring context for disambiguation.
  • Syntactic Complexity: The mixing of Kannada's SOV order with English's SVO order creates a "third grammar" that complicates traditional parsing.

Conclusion

This work provides the first annotated corpus for POS tagging of code-mixed Kannada-English social media data. While the current dataset is a significant foundation, future work aims to expand the corpus size and include additional layers of annotation, such as Named Entity Recognition (NER). Additionally, exploring code-mixing involving native Kannada script alongside Roman transliteration remains an important avenue for future research.

Summary updated

Abstract

Part-of-Speech (POS) is one of the essential tasks for many Natural Language Processing (NLP) applications. There has been a significant amount of work done in POS tagging for resource-rich languages. POS tagging is an essential phase of text analysis in understanding the semantics and context of language. These tags are useful for higher-level tasks such as building parse trees, which can be used for Named Entity Recognition, Coreference resolution, Sentiment Analysis, and Question Answering. There has been work done on code-mixed social media corpus but not on POS tagging of Kannada-English code-mixed data. Here, we present Kannada-English code- mixed social media corpus annotated with corresponding POS tags. We also experimented with machine learning classification models CRF, Bi-LSTM, and Bi-LSTM-CRF models on our corpus.
BibTeX
@inproceedings{reddy2020creation,
  author = {Appidi Abhinav Reddy and Vamshi Krishna Srirangam and Darsi Suhas and Manish Shrivastava},
  title = {{Creation of Corpus and Analysis in Code-Mixed Kannada-English Social Media Data for POS Tagging}},
  booktitle = {ICON},
  year = {2020},
  url = {https://openreview.net/forum?id=wMvqFK7JVA},
}
Figure