TL;DR
This paper tackles the challenge of distinguishing between Romanized Hindi and Urdu, languages often conflated when written in the Latin script. They leverage large language models (LLMs) to generate synthetic training data, demonstrating that even a simple decision tree classifier can achieve high accuracy on this task, outperforming more complex neural models on out-of-domain data.
What Problem Does This Paper Address?
Hindi and Urdu, while distinct in their native scripts (Devanagari and Perso-Arabic, respectively), are highly mutually intelligible in spoken form. When written in the Latin script (romanization), their differences become even more subtle, leading to their frequent conflation in language identification systems and datasets. This is problematic because, despite their linguistic similarities, Hindi and Urdu represent different cultural contexts, and models trained on conflated data may fail to provide culturally appropriate predictions, particularly for Urdu speakers who may be a minority in many datasets. The paper aims to bridge this gap by developing methods to reliably distinguish between romanized Hindi and Urdu.
Methods
The core of the paper's methodology revolves around using pre-trained Large Language Models (LLMs) to generate synthetic training data. Specifically, they fine-tune the mT5 model on the "hi-Latn" portion of the mC4 corpus. They then create prompts for this fine-tuned model by embedding seed words (words known to be indicative of either Hindi or Urdu) into generic sentence templates. The LLM is then prompted to fill in the remaining parts of the sentence. This generated text, labeled according to the seed word used in the prompt, serves as training data for a language identification classifier. They compare two types of classifiers: a simple decision tree using character n-grams and a fine-tuned mT5 model acting as a classifier.
Datasets
- Dakshina: A dataset containing hand-romanized sentences from Hindi and Urdu Wikipedia articles, specifically designed to distinguish between the two languages. This dataset is used for evaluation and as a "topline" for performance comparison.
- mC4: A large, web-scraped multilingual corpus. The "hi-Latn" partition, nominally for romanized Hindi, is hypothesized to contain a mix of Hindi and Urdu. This dataset is used for creating a baseline classifier and as a source for LLM fine-tuning.
- Seed words: A collection of 147 Hindi/Urdu word pairs that differ between the languages but share the same meaning. These are crucial for generating labeled synthetic data.
Key Insights and Ideas
- LLMs as data generators: The paper demonstrates that LLMs can be effectively used to generate labeled training data for tasks where obtaining such data is difficult, even when the distinction is subtle and implicit in the pretraining data.
- Simple classifiers can be powerful: Surprisingly, a simple decision tree classifier, when trained on LLM-generated data, performs remarkably well, even outperforming a more complex neural classifier on out-of-domain data. This suggests that focusing on local, word-form features can be highly effective.
- Overfitting in neural models: The more powerful mT5 classifier, when trained on LLM-generated data, tends to overfit, performing at chance on the development set. This highlights the importance of carefully considering the training data and model complexity.
- Implicit knowledge in LLMs: The success of the decision tree classifier trained on generated data suggests that LLMs implicitly capture subtle linguistic differences between romanized Hindi and Urdu, which can be exploited for language identification.
Key Results
- A decision tree classifier trained on LLM-generated data achieved 83.4% accuracy on the Dakshina development set, approaching the topline performance of 85.6% achieved by a decision tree trained on the matched Dakshina data.
- The mT5 classifier, when trained on LLM-generated data, achieved 49.2% accuracy, performing at chance, while it achieved 96.7% accuracy on the matched Dakshina topline data.
- A baseline classifier trained on mC4 sentences performed at chance for both decision tree and mT5 models.
- The authors estimate that 61.1% of the mC4's romanized Hindi data might actually be romanized Urdu, based on their model's performance.
Limitations and Future Work
- Tentative conclusions: The authors acknowledge that some conclusions about the connection between training data and outcomes are tentative due to the nature of the LLM input and the lack of direct control over the pretraining data.
- Bias in estimation: The estimated prevalence of Urdu in mC4 (61.1%) is considered a potential overestimate, suggesting a possible bias towards Urdu in their model. Further validation is needed.
- Future work: The authors propose manual validation and error analysis of classifier performance, applying the method to language varieties with well-documented variations (e.g., US vs. UK English), and investigating similar confounds in other language families (e.g., Bosnian, Croatian, Montenegrin). They also suggest exploring other LLMs and generation strategies.
Summary updated