Labeling of Query Words using Conditional Random Field

Satanu Ghosh, Souvick Ghosh, Dipankar DasarXiv 2016 · arXiv

TL;DR

This paper presents a Conditional Random Field (CRF) based approach for word-level language identification in multi-lingual search queries written in Roman script. By leveraging contextual features and language-specific lexicons, the system achieves a 75.5% token-level accuracy across English and eight transliterated Indian languages.

What Problem Does This Paper Address?

In the era of social media, users frequently engage in code-switching and code-mixing—blending multiple languages within a single sentence or query. For Indian languages, this is further complicated by the use of the Roman script (transliteration) instead of native scripts like Devanagari or Bengali.

Standard language identification tools often fail at the word level because:

  1. Ambiguity: A word like "take" is a common English verb, but its transliterated form is also a valid Bengali word.
  2. Informal Text: Social media queries are rife with phonetic typing (e.g., "gr8" for "great"), non-standard grammar, and random contractions.
  3. Script Uniformity: When everything is in Roman script, you lose the immediate orthographic clues provided by different alphabets.

The authors tackle the "Query Word Labeling" task from the FIRE 2015 workshop, aiming to label every token in a query as English, one of eight regional Indian languages, a Named Entity (NE), or a special symbol (X).

Methods

The researchers employed a Conditional Random Field (CRF) framework, specifically using the CRF++ toolkit. CRFs are well-suited for sequence labeling because they consider the state of neighboring tokens rather than treating each word in isolation.

The core of their approach lies in meticulous feature engineering:

  • Contextual Features: They used a sliding window of the three previous and three next words. This is vital for resolving ambiguity; the language of "take" is determined by the words surrounding it.
  • Dictionary/Lexicon Features: They integrated nine different language-specific word lists, including resources like the Corncob Web Dictionary for English and Google Input Tools for transliterated Indian languages.
  • Orthographic Features: Boolean flags for capitalization (useful for Named Entities), digits (for phonetic shortcuts like "4nds"), and special characters (for hashtags and emoticons).
  • Named Entity Recognition: They utilized the Stanford NE Tagger combined with a custom-built list of 648 distinct names.

Datasets

The study used the dataset provided by the FIRE 2015 organizers:

  • Training Set: 2,908 sentences.
  • Test Set: 792 sentences.
  • External Resources:
    • English and Bengali word frequency lists.
  • Hindi word transliteration pairs (30,823 entries).
  • Gujarati word transliteration pairs (a smaller set of 546 entries).
  • Google Input Tools lookup tables for all 8 Indian languages.

Key Insights and Ideas

  • The Power of Context: The authors highlight that words like take, are, pore, and bad are "truly ambiguous" as they exist in both English and Bengali. The CRF's ability to look at the "dynamic unigram feature" (the language labels of neighbors) was the primary mechanism for resolving these overlaps.
  • Word Length Matters: A simple but effective feature was token length. The authors noted that words in Indian languages tend to be longer than English words when transliterated, providing a subtle but useful statistical signal.
  • Script-Specific Challenges: The system performed significantly better on languages with larger available resources (English, Hindi, Bengali) compared to those with sparse data (Gujarati, Kannada), highlighting the "data hunger" of dictionary-based features.

Key Results

The system achieved an overall token-level accuracy of 75.49%. However, the performance varied wildly across different categories.

Table 1: Tokens level Results for language identification

As seen in Table 1, the F-Measures for English (0.892), Hindi (0.797), and Bengali (0.748) were robust. In contrast, Gujarati suffered significantly with an F-Measure of only 0.1279, likely due to the very small lexicon (546 entries) available during training.

Table 2: Other performance metrics

While token-level accuracy was decent, the Utterance Accuracy (the percentage of entire sentences labeled correctly) was much lower at 21.59%, illustrating how a single error in a short query can cause the whole sequence to fail.

Limitations and Future Work

The authors identified several areas for improvement:

  • Vocabulary Overlap: Many Indian language words are wrongly classified as English due to overlapping vocabularies.
  • Named Entity Recognition: The sub-classification of NEs was a weak point. The authors suggest that a dedicated NE-classification module could significantly reduce these errors.
  • Feature Engineering: Interestingly, incorporating character n-grams (n=1 to 5) actually caused performance to decline, a counter-intuitive result that warrants further investigation.
  • Post-Processing: The current system lacks a post-processing heuristic to "clean up" unlikely sequences of language labels, which could improve the utterance-level accuracy.

Summary updated

Abstract

This paper describes our approach on Query Word Labeling as an attempt in the shared task on Mixed Script Information Retrieval at Forum for Information Retrieval Evaluation (FIRE) 2015. The query is written in Roman script and the words were in English or transliterated from Indian regional languages. A total of eight Indian languages were present in addition to English. We also identified the Named Entities and special symbols as part of our task. A CRF based machine learning framework was used for labeling the individual words with their corresponding language labels. We used a dictionary based approach for language identification. We also took into account the context of the word while identifying the language. Our system demonstrated an overall accuracy of 75.5% for token level language identification. The strict F-measure scores for the identification of token level language labels for Bengali, English and Hindi are 0.7486, 0.892 and 0.7972 respectively. The overall weighted F-measure of our system was 0.7498.
BibTeX
@misc{ghosh2016labeling,
  author = {Satanu Ghosh and Souvick Ghosh and Dipankar Das},
  title = {{Labeling of Query Words using Conditional Random Field}},
  year = {2016},
  eprint = {1607.08883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1607.08883},
}
Figure