Complexity Metric for Code-Mixed Social Media Text

Souvick Ghosh, Satanu Ghosh, Dipankar DasarXiv 2017 · arXiv

TL;DR

This paper introduces the Complexity Factor (CF), a new metric designed to quantify how difficult "code-mixed" social media text (text blending multiple languages) is for Natural Language Processing (NLP) systems. By accounting for the number of languages and the frequency of switching between them, CF provides a more nuanced measurement than previous indices, helping researchers better evaluate the difficulty of multilingual datasets.

What Problem Does This Paper Address?

In multilingual societies like India, social media users frequently engage in code-mixing—switching between languages (e.g., English, Hindi, and Bengali) within a single post or even a single sentence. Furthermore, these users often transliterate their native languages into the Roman script for convenience.

For NLP researchers, this creates a "noisy" environment. Standard tools for sentiment analysis or part-of-speech tagging often fail when faced with these linguistic shifts. To improve these systems, we first need a way to measure exactly how "complex" a piece of text is. The existing standard, the Code Mixing Index (CMI), only looks at the ratio of words from different languages. It ignores how those words are arranged—for example, it treats a sentence that switches languages once the same as one that switches back and forth every other word. This paper aims to fill that gap.

Methods

The authors propose the Complexity Factor (CF), which integrates three distinct linguistic dimensions:

  1. Language Factor (LF): The ratio of total words to the number of distinct languages. A lower LF indicates a more multilingual (and thus complex) sentence.
  2. Switching Factor (SF): The ratio of actual language switch points to the maximum possible switch points in a sentence. This captures the "jitter" or frequency of transitions.
  3. Mix Factor (MF): Based on the original CMI, this represents the fraction of words belonging to non-dominant languages.

The authors combined these into a single score ($CF = \frac{a \cdot MF + b \cdot SF}{f(LF)}$) and tested three variants:

  • CF1: A simple baseline using raw LF.
  • CF2: Uses a linear function to dampen the impact of the language factor.
  • CF3: Uses a geometric function (arctan) for dampening.

These dampening functions are crucial because social media text rarely exceeds three languages; without dampening, the language count would disproportionately skew the complexity score.

Datasets

The study utilized two primary corpora to validate the new metric:

  • FIRE 2015 Shared Task Corpus: A highly multilingual dataset containing 9 Indic languages (Bengali, Hindi, Tamil, etc.) plus English. It consists of 3,701 sentences and over 63,000 tokens.
  • ICON 2015 Shared Task Corpus: A trilingual dataset focused on Bengali, Hindi, and English, collected from Facebook chat groups like "JU Confession."
Statistics of the highly multilingual FIRE 2015 Corpus
Statistics of the trilingual ICON 2015 Corpus

Key Insights and Ideas

  • Switching Frequency Matters: The most significant "aha moment" is that the arrangement of words is just as important as the count. A sentence that alternates languages word-by-word is significantly harder for a machine to process than one that switches once in the middle. CF captures this, whereas CMI does not.
  • The "Social" vs. "Competence" Switch: The authors highlight that code-switching isn't just about a lack of vocabulary; it's often a social marker to identify with a group or a conversational tool to add emphasis.
  • Transliteration Complexity: The paper notes that the use of Roman script for Indic languages adds a layer of complexity that necessitates these specific metrics, as it removes the script-based cues usually used for language identification.

Key Results

The authors demonstrated that their CF2 and CF3 metrics provided a much wider and more accurate range of complexity scores compared to CMI.

  • Mathematical Precision: In theoretical cases where a sentence switches languages every word, CMI gave a score of 90, while CF2/CF3 reached 95, better reflecting the extreme difficulty.
  • Corpus Comparison: The FIRE corpus was found to be significantly more complex than the ICON corpus. The average CF2 for FIRE was 10.54, compared to only 4.83 for ICON.
  • Sensitivity to Switches: In real-world examples (Case 6), a sentence with frequent switches received a CF2 of 64.22, while CMI only gave it a 40, failing to account for the high "Switching Factor."
Graph showing the distribution of CF2 complexity scores relative to sentence length for the FIRE corpus.

Limitations and Future Work

While the Complexity Factor is a step forward, the authors acknowledge several areas for improvement:

  • Intra-word Mixing: The current metric does not account for complexity at the morphological level (e.g., an English root word with a Bengali suffix).
  • Broader Validation: The index needs to be tested on non-Indic language pairs, such as Spanish-English or Mandarin-English.
  • Downstream Correlation: Future research should explicitly measure how well CF scores correlate with the actual drop in accuracy for specific NLP tasks like sentiment analysis or machine translation.

Summary updated

Abstract

An evaluation metric is an absolute necessity for measuring the performance of any system and complexity of any data. In this paper, we have discussed how to determine the level of complexity of code-mixed social media texts that are growing rapidly due to multilingual interference. In general, texts written in multiple languages are often hard to comprehend and analyze. At the same time, in order to meet the demands of analysis, it is also necessary to determine the complexity of a particular document or a text segment. Thus, in the present paper, we have discussed the existing metrics for determining the code-mixing complexity of a corpus, their advantages, and shortcomings as well as proposed several improvements on the existing metrics. The new index better reflects the variety and complexity of a multilingual document. Also, the index can be applied to a sentence and seamlessly extended to a paragraph or an entire document. We have employed two existing code-mixed corpora to suit the requirements of our study.
BibTeX
@misc{ghosh2017complexity,
  author = {Souvick Ghosh and Satanu Ghosh and Dipankar Das},
  title = {{Complexity Metric for Code-Mixed Social Media Text}},
  year = {2017},
  eprint = {1707.01183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.01183},
}

Loading related content...

Figure