LexiLogic@CALCS 2025: Predicting Preferences in Generated Code-Switched Text

Pranav Gupta, Souvik Bhattacharyya, Niranjan Kumar M, Billodal RoyNAACL 2025

TL;DR

This paper tackles the challenge of evaluating code-switched text generation, which is crucial for building better NLP systems for multilingual communities. The authors explore various methods, including fine-tuning GPT-2 and using logistic regression on embeddings, to predict human preferences for generated code-switched text, ultimately finding that large models like GPT-4o perform best.

What Problem Does This Paper Address?

Code-switching, the practice of alternating between languages within a conversation, is a natural phenomenon for many multilingual individuals. While NLP has made strides in monolingual text generation, generating and evaluating code-switched text remains a significant challenge. Existing evaluation metrics, often designed for monolingual text, may not adequately capture the nuances of code-switched language. This paper addresses the need for better methods to predict human preferences for code-switched text, which is a critical step towards improving and optimizing code-switched generation systems. This is particularly important for languages and domains that are less represented in pre-training datasets.

Methods

The authors experimented with several approaches to predict human preferences for code-switched text:

  • Fine-tuning GPT-2 as a Reward Model: They adapted the GPT-2 model by adding a linear head to output a scalar reward score. This model was trained to maximize the reward for the preferred code-switched completion, using a cross-entropy loss function inspired by the Bradley-Terry model. They also explored cross-lingual transfer by training on different combinations of language pairs.
  • Logistic Regression on Multilingual Embeddings: This method involved training a 3-class logistic regression model on top of multilingual embeddings generated by Cohere's embed-multilingual-v3.0 model. The embeddings were derived from concatenations of original and generated sentences.
  • FastText Classification: A standard and efficient text classification algorithm, FastText, was used to train a 3-class classifier on concatenated text inputs.
  • GPT-4o for Preference Prediction: Leveraging the reported high correlation of GPT-4o with human judgment for code-mixed text, the authors used it as a final judge to predict preferences, explaining the task and providing an approximate prior distribution of preferences. They optimized batch sizes for inference efficiency.

Datasets

The primary dataset used in this study is the labeled component of the CSPref dataset (Kuwanto et al., 2024). This dataset contains pairs of code-switched generations and human judgments on which generation is preferred. The authors split this dataset into training and testing sets, carefully ensuring no leakage by splitting based on unique original sentence pairs. The dataset covers three language pairs: English-Malayalam, English-Tamil, and English-Hindi.

Key Insights and Ideas

  • Cross-lingual Transfer is Beneficial: Training the GPT-2 reward model on multiple language pairs led to improved performance across all pairs, suggesting that learning from one language pair can positively impact performance on others.
  • Larger Models Excel at Preference Prediction: The study strongly indicates that larger, more capable models like GPT-4o are superior for predicting human preferences in code-switched text compared to smaller models or traditional machine learning approaches.
  • Human Judgment is Complex: The paper highlights that predicting human preferences is challenging, and even confident model predictions can fluctuate, especially in cases of ties, as visualized in Figure 1.
  • Data Quality Matters: Exploratory analysis suggested that some generated sentences might lack coherence due to their origin from smaller LLMs, impacting the quality of preference judgments.

Key Results

  • GPT-4o Achieved the Highest Accuracy: GPT-4o demonstrated the best performance among all the models tested, achieving a test set accuracy of 0.66.
  • Fine-tuned GPT-2 Performance: The fine-tuned GPT-2 model achieved a test set accuracy of 0.53. Performance improved when trained on more language pairs.
  • Logistic Regression and FastText Baseline: The logistic regression model on Cohere embeddings achieved 0.52 accuracy, while FastText achieved 0.38.
  • Table 3 shows detailed accuracy breakdowns for GPT-2 across different training configurations. For instance, when trained only on English-Hindi, it achieved 0.47 accuracy on English-Hindi test data, but lower accuracies on English-Tamil (0.41) and English-Malayalam (0.46).

Limitations and Future Work

  • Dataset Size and Diversity: The authors acknowledge that a larger training set with a more diverse collection of language pairs could further improve the ability of models to capture human preferences.
  • Ethical Implications: The paper raises concerns about the ethical implications of using preference prediction models in real-world applications, particularly regarding potential biases and the disenfranchisement of communities with unique code-switching patterns.
  • Future Directions: The authors suggest exploring larger models and datasets, conducting deeper comparative analyses across languages, and investigating prompt optimization techniques (e.g., DSPy) and parameter-efficient fine-tuning methods (e.g., LoRA). They also propose exploring the use of these models as reward functions for aligning LLMs to generate more natural code-mixed text.
  • Metric Benchmarking: The paper questions whether correlations between automated metrics and human judgment are a sufficient benchmark for comparing models.

Summary updated

BibTeX
@inproceedings{gupta2025lexilogic,
  author = {Pranav Gupta and Souvik Bhattacharyya and Niranjan Kumar M and Billodal Roy},
  title = {{LexiLogic@CALCS 2025: Predicting Preferences in Generated Code-Switched Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure