Comparing Grammatical Theories of Code-Mixing

Adithya Pratapa, Monojit ChoudhuryW-NUT 2021 · OpenReview

Sure, here's a blog-style summary of the paper "Comparing Grammatical Theories of Code-Mixing":

TL;DR

This paper tackles the challenge of generating realistic code-mixed text by comparing two prominent grammatical theories: Matrix-Embedded Language (ML) and Equivalence Constraint (EC). Through a large-scale human evaluation, they found that both grammatical theories significantly outperform heuristic-based models, and surprisingly, neither theory emerged as a clear winner in terms of perceived naturalness.

What Problem Does This Paper Address?

Code-mixing, the practice of blending multiple languages within a single conversation, is increasingly prevalent in informal text, especially on social media. This phenomenon poses a significant challenge for Natural Language Processing (NLP) tasks like speech recognition, translation, and dialogue systems, which often struggle to accurately model and generate such mixed-language content. While various approaches exist, including neural sequence-to-sequence models and rule-based systems derived from linguistic theories, there's a lack of comprehensive comparison between these grammatical theories themselves. This paper aims to bridge that gap by evaluating the effectiveness of two major code-mixing theories, ML and EC, against each other and against simpler heuristic methods.

Methods

The authors compare two main types of models for generating code-mixed text:

  • Grammatical Models:
    • Equivalence Constraint (EC): This theory, proposed by Poplack (1980), posits that languages in a code-mixed sentence must adhere to a shared grammatical structure. Any divergence is resolved by "node collapse" to maintain equivalence.
  • Matrix-Embedded Language (ML): Myers-Scotton's (1993) theory suggests that a "matrix" language can accommodate constituents from an "embedded" language, with specific restrictions on certain grammatical elements (like verbs, determiners, etc.) and nesting.
  • Relaxed Variants: To understand the impact of specific constraints, the authors also tested relaxed versions of ML and EC, allowing for more flexibility in mixing.
  • Heuristic Models: These models do not rely on deep linguistic theories but instead use simpler approaches:
  • Aligned (Haligned): Generates code-mixed text by sampling words from aligned parallel sentences.
  • Dictionary (Hdict): Uses a bilingual dictionary to substitute words from a monolingual sentence with their counterparts in another language.
  • Parallel (Hparallel): Assumes a linear mapping between word orders in parallel sentences, useful when explicit alignments are missing.

The core of their evaluation involved a large-scale human evaluation conducted on Amazon Mechanical Turk. They collected judgments on the understandability and naturalness of code-mixed sentences generated by these models.

Datasets

The study utilized Spanglish (English-Spanish) sentences for their evaluation. The data was prepared by:

  • Collecting monolingual tweets in English and Spanish.
  • Translating these tweets to the other language using the Microsoft Translator API to create parallel sentences.
  • Extracting word-level alignments using the fast_align toolkit.
  • Generating constituent parses using the Stanford PCFG parser for the grammatical models.

The paper does not introduce a new benchmark dataset but rather uses existing data preparation techniques to generate synthetic code-mixed sentences for evaluation.

Key Insights and Ideas

  • Grammar Matters: The study strongly suggests that linguistic theories, even with their inherent complexity, are crucial for generating more understandable and natural-sounding code-mixed text compared to simpler heuristic approaches.
  • No Clear Winner Between Theories: Surprisingly, the human judges did not find one grammatical theory (ML or EC) significantly better than the other in terms of naturalness. This implies that both theoretical frameworks capture important aspects of code-mixing.
  • Relaxation Doesn't Always Improve: The original ML theory performed better than its relaxed variants, indicating that its specific constraints might be more beneficial than initially thought. For EC, the relaxed version showed no clear improvement.
  • Sentence Length and Switch Points: Human judges preferred shorter sentences and fewer language switch points, aligning with observations of natural code-mixing in real-world data.
  • Translation Quality is Key: Higher pseudo fuzzy match scores (indicating better translation quality of the source data) correlated with better perceived quality of the generated code-mixed text.

Key Results

  • Grammatical models significantly outperform heuristic models in terms of human judgments on understandability and naturalness.
  • The original ML theory was found to be better than its relaxed variants, suggesting the importance of its specific constraints.
  • There was no significant preference between the original ML and EC theories.
  • Human judges preferred fewer switch points in code-mixed texts.
  • Sentences with higher translation quality (PFMS) were rated better.
  • Speakers preferred shorter sentences.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Focus on Grammatical Models: The study primarily focused on grammatical theories and did not compare them against neural sequence-to-sequence models, which represent another significant line of research in code-mixed text generation.
  • Limited Language Pair: The evaluation was conducted on Spanglish (English-Spanish) only. Extending the analysis to other language pairs would provide broader insights.
  • Future Work: The authors suggest extending the naturalness evaluation to sequence-to-sequence models and expanding the analysis to other language pairs. They also mention that the Functional Head Constraint (FHC) theory was not evaluated due to the unavailability of a computational model.

Summary updated

BibTeX
@inproceedings{pratapa2021comparing,
  author = {Adithya Pratapa and Monojit Choudhury},
  title = {{Comparing Grammatical Theories of Code-Mixing}},
  booktitle = {W-NUT},
  year = {2021},
  url = {https://openreview.net/forum?id=R3Ql5nZN7MM},
}
Figure