Labeling the Languages of Words in Mixed-Language Documents using Weakly Supervised Methods

Ben King, Steven AbneyNAACL 2013

TL;DR

This paper tackles the challenge of identifying the language of individual words within documents that mix multiple languages. They propose a weakly supervised sequence labeling approach using conditional random fields (CRFs) trained with generalized expectation criteria, achieving state-of-the-art results and outperforming simpler methods. This is crucial for building language resources for minority languages and for accurately processing the increasingly multilingual web.

What Problem Does This Paper Address?

Language identification is a well-studied problem, but typically focuses on classifying an entire document's language. However, many real-world documents, especially those related to minority languages or from the web, contain a mix of languages at the word or sentence level. This "code-switching" or multilingualism poses a significant challenge for existing tools. For instance, efforts to build corpora for minority languages often fail because the collected web pages also contain dominant languages like English. This paper specifically addresses the granular problem of identifying the language of each word in a mixed-language document, a task that has received little attention. They aim to develop methods that can handle this without requiring fully labeled multilingual data for training.

Methods

The authors frame the problem as a sequence labeling task. They explore several weakly supervised learning methods, focusing on those that can learn from limited, monolingual training data.

  • Independent Word Classification (Baseline): As a baseline, they first evaluate classifying each word independently using various classifiers (logistic regression, naive Bayes, decision tree, Winnow2) and features (n-grams, full words). This ignores sequential dependencies.
  • Conditional Random Field (CRF) with Generalized Expectation (GE) Criteria: This is their primary proposed method. CRFs are powerful for sequence labeling. The "weakly supervised" aspect comes from using Generalized Expectation criteria. Instead of relying on fully labeled sequences, GE criteria allow the model to be biased by "soft constraints" derived from monolingual data and independent word predictions. They use a KL-divergence term to align the model's expected output with these constraints.
  • Hidden Markov Model (HMM) with Expectation Maximization (EM): They also implement a generative HMM trained using the EM algorithm. This approach uses monolingual data to estimate emission probabilities (word-to-language likelihoods) and then iteratively refines transition probabilities (language-to-language likelihoods) using the document's own predicted labels.
  • Logistic Regression with Generalized Expectation: Similar to the CRF approach, they apply GE criteria to logistic regression models, a form of label regularization.

Crucially, they introduce new features for the sequence labeling models that are not applicable to independent word classification, such as features for non-word characters (punctuation, digits) between words.

Datasets

  • Evaluation Data: The authors created a new corpus of over 250,000 words from mixed-language web documents. They specifically selected documents containing exactly two languages (one being English, the other a minority language with Latin orthography) to simplify annotation and evaluation.
  • Training Data: For weakly supervised training, they used small monolingual samples from 643 languages sourced from the Universal Declaration of Human Rights, non-English Wikipedias, Jehovah's Witnesses website, and the Rosetta project. For experiments, they used samples from 30 of these languages, with sizes varying from 92 to over 16,000 words. To mitigate sample size variations, they sampled words with replacement.

Key Insights and Ideas

  • Word-level is Crucial: Sentence-level language identification is too coarse. Even if sentences are correctly labeled, word-level accuracy is significantly lower.
  • Context Matters: Independent word classification is outperformed by sequence labeling methods because word context (surrounding words, punctuation) provides valuable clues about language.
  • Weak Supervision is Viable: The GE criteria approach demonstrates that effective language identification can be achieved with limited, monolingual training data, making it suitable for low-resource languages.
  • CRFs Shine: Conditional Random Fields, especially when trained with Generalized Expectation, emerge as the most robust and accurate method for this task.
  • Named Entities are Tricky: Named entities (proper nouns, abbreviations) pose a significant challenge, as they often don't strictly follow the language of their context and are difficult for models to handle without specific rules.

Key Results

  • The GE-trained CRF model achieved the highest accuracy across various training data sizes.
  • With only 10 words of each language as training data, the GE-trained CRF correctly labeled 88% of words.
  • The best independent word classifier (naïve Bayes) achieved around 88% accuracy with 1000 sampled words, while the GE-trained CRF reached similar accuracy with significantly less data (around 100-200 words).
  • Error analysis showed that named entities were the most frequent error type (around 40-50% of errors for top models), followed by "shared words" (words that could belong to either language).

Limitations and Future Work

  • Named Entity Handling: The current annotation rules for named entities were arbitrary and not well-handled by the models. Future work could involve creating a separate label for named entities or integrating language-independent named entity recognition systems.
  • A Priori Language Set: The current method requires knowing the set of possible languages beforehand. For documents with potentially thousands of languages, this is impractical.
  • Two-Step Approach: To address the above, the authors propose a future two-step approach: first, identify the languages present in a document with high accuracy, and second, use this identified set to label words within that document, lifting the two-language restriction.
  • Constraint Estimation: The accuracy of the GE-trained CRF is sensitive to the estimation of marginal label distributions. Future work could explore pruning constraints or adjusting GE-criteria coefficients to improve robustness.

Summary updated

BibTeX
@inproceedings{king2013labeling,
  author = {Ben King and Steven Abney},
  title = {{Labeling the Languages of Words in Mixed-Language Documents using Weakly Supervised Methods}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure