Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages

Olga Kellert, Nemika Tyagi, Muhammad Imran, Nelvin Licona-Guevara, Carlos Gómez-RodríguezEMNLP 2025

TL;DR

This paper introduces "BiLingua Pipeline," a novel LLM-based approach for automatically generating Universal Dependencies (UD) annotations for complex code-switched (CSW) text, especially in low-resource languages. By leveraging GPT-4.1 with carefully crafted prompts and expert review, they achieve high accuracy and significantly outperform existing parsers, paving the way for better linguistic analysis of multilingual communication.

What Problem Does This Paper Address?

Code-switching (CSW), the phenomenon of mixing languages within a single conversation or sentence, is incredibly common but poses a significant challenge for natural language processing (NLP), particularly for syntactic parsing. Existing NLP models, trained primarily on monolingual data, struggle to accurately analyze the mixed grammatical structures found in CSW. This is especially problematic for low-resource languages where annotated data is scarce, hindering research and development in understanding multilingual communication. While some prior work has explored LLMs for sequence-level tagging in CSW, a systematic approach to capturing complex syntactic structures and generating high-quality, linguistically validated annotations has been missing.

Methods

The core of this work is the BiLingua Pipeline, which uses a large language model (LLM), specifically GPT-4.1, to generate Universal Dependencies (UD) style syntactic annotations for code-switched sentences. The process involves:

  • Few-Shot LLM Prompting: The LLM is guided by carefully designed prompts that include examples and specific instructions for handling CSW phenomena like contractions, repetitions, ellipsis, and punctuation. This approach allows the LLM to learn and apply UD conventions without extensive supervised training data.
  • Expert Review and Validation: To ensure accuracy and linguistic plausibility, the LLM-generated annotations are reviewed and corrected by native speakers. This human-in-the-loop process is crucial for refining the annotations and creating reliable datasets.
  • Semantically Similar Tag Grouping: For evaluation, the authors group semantically similar UD tags. This acknowledges that linguistic analysis can have plausible variations and prevents penalizing linguistically motivated alternatives, providing a more realistic benchmark than strict LAS scores alone.
  • Syntactic Analysis of Switch Points: The pipeline enables detailed analysis of where language switches occur within sentences, examining the syntactic roles (e.g., subjects, objects) and dependency relations involved.

Datasets

The authors utilize and extend existing datasets:

  • Miami Corpus (Spanish-English): A well-known dataset of spoken Spanish-English CSW interactions, used for developing and evaluating the pipeline.
  • GUA-SPA Dataset (Spanish-Guaraní): A dataset of Spanish-Guaraní CSW utterances from social media and news, representing a low-resource language pair.
  • New Spanish-English and Spanish-Guaraní UD-Annotated CSW Datasets: The paper introduces newly created, expert-reviewed UD-annotated datasets derived from the above sources, including the first UD-parsed corpus for Spanish-Guaraní CSW.

Key Insights and Ideas

  • LLMs as Powerful Bootstrapping Tools: The research demonstrates that LLMs, when guided by expert knowledge and careful prompting, can effectively bootstrap syntactic resources for complex, low-resource CSW scenarios, significantly reducing the need for extensive manual annotation.
  • Syntactic Roles Matter More Than POS: Analyzing dependency relations at switch points reveals more nuanced insights into CSW behavior than traditional part-of-speech tagging alone. For instance, syntactic subjects (nsubj) are identified as particularly frequent switch points.
  • Flexibility in Low-Resource CSW: The Spanish-Guaraní data shows greater syntactic flexibility at switch points compared to Spanish-English, suggesting that typologically diverse language pairs might exhibit different patterns of code-switching.
  • Emoji Usage Licenses More Syntactic Fluidity: In the Spanish-Guaraní dataset, messages containing emojis exhibit more frequent switching at discourse-sensitive roles and verb heads, indicating that informal communication styles and pragmatic context can influence syntactic flexibility.
  • Current Evaluation Metrics Are Too Rigid: The authors argue that standard UD evaluation metrics (like LAS) are too strict for complex, multilingual data and that a more flexible framework, acknowledging semantic similarity in tags, is needed.

Key Results

  • The BiLingua Pipeline achieves up to 95.29% LAS (after expert review) on Spanish-English CSW data, significantly outperforming prior baselines (which reported LAS below 75%).
  • For the low-resource Spanish-Guaraní dataset, the pipeline achieved 77.42% LAS after expert review.
  • A general-purpose multilingual parser (UDSL) achieved only 14.71% LAS on the Spanish-English CSW data, highlighting the limitations of off-the-shelf models for this task.
  • UPOS tagging accuracy remained high (99.54% for Spanish-English, 84.21% for Spanish-Guaraní) even when dependency parsing was slightly lower.
  • The analysis of switch points in Spanish-English revealed that nsubj (nominal subject) and det (determiner) are common switch sites.

Limitations and Future Work

  • Inconsistent LLM Behavior: LLMs can still be inconsistent in applying rules for repetitions and ellipsis, and their handling of functional verbs can vary, impacting the reliability of syntactic generalizations.
  • Morphological Complexity in Low-Resource Languages: For agglutinative languages like Guaraní, complex tokens may need morphological analysis or splitting for more accurate parsing.
  • UD Framework Complexity: The complexity of UD conventions can pose a challenge for annotators unfamiliar with formal linguistic parsing, requiring detailed guidelines and training.
  • Future Work: The authors suggest incorporating language-specific morphological analyzers, improving token-splitting mechanisms, and developing more flexible evaluation frameworks that account for linguistic variation. They also highlight the need for careful deployment of LLMs to avoid propagating biases and to ensure linguistic expertise is integrated.

Summary updated

BibTeX
@inproceedings{kellert2025parsing,
  author = {Olga Kellert and Nemika Tyagi and Muhammad Imran and Nelvin Licona-Guevara and Carlos Gómez-Rodríguez},
  title = {{Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure