TL;DR
This paper investigates how using alternative text representations, specifically ISO-15919 and a custom IndicSOUNDEX, can improve Natural Language Understanding (NLU) models for Indic languages, which often feature code-switching and linguistic variations. The study demonstrates that these alternate representations significantly boost NLU performance, particularly when training data is scarce.
What Problem Does This Paper Address?
Voice assistants rely heavily on Natural Language Understanding (NLU) models to interpret user requests. However, NLU models face significant challenges in languages that exhibit code-switching (mixing languages within a sentence) and variations in spelling and word forms. These inconsistencies, often introduced by Automatic Speech Recognition (ASR) systems, can lead to misinterpretations and reduced accuracy. This is particularly prevalent in Indic languages, where code-switching is common. The paper aims to address this by exploring whether alternative text representations can make NLU models more robust to these challenges, especially in low-resource scenarios.
Methods
The researchers propose using two alternate representations of input text for NLU models:
- ISO-15919: A standardized Latin-based transliteration system for Indic languages and scripts. This aims to provide a common representation for characters across different Indic scripts.
- IndicSOUNDEX: A custom phonetic-like algorithm designed to work for a range of Indic languages. Similar to traditional SOUNDEX, it aims to group words that sound alike, thus mitigating phonetic variations and misspellings.
They developed a deep neural network (DNN) based multi-task learning framework to predict both Intent Classification (IC) and Named Entity Recognition (NER). This framework incorporates the original text embeddings alongside embeddings derived from these alternate representations. They experimented with a baseline model (original text only) and two candidate models, one for each alternate representation.
Datasets
The study utilized datasets from four Indic languages:
- Hindi: A large-scale, internal real-world dataset.
- Tamil, Marathi, Telugu: Smaller datasets collected and annotated by third-party vendors.
These datasets were partitioned into training, validation, and testing sets, with specific splits for the high-resource Hindi language and the low-resource Tamil, Marathi, and Telugu languages. All datasets contained examples of code-switching.
Key Insights and Ideas
- Low-resource languages benefit most: The most significant improvements in NLU performance were observed in low-resource languages (Tamil, Marathi, Telugu), suggesting that alternate representations are particularly valuable when training data is limited.
- Code-switching and variations are key challenges: The paper highlights how code-switching and orthographic/morphological variations pose substantial hurdles for NLU models, especially in languages like Hindi.
- Phonetic similarity matters: IndicSOUNDEX, by capturing phonetic similarities, proved effective in mitigating some of the ASR-induced errors and linguistic variations.
- Standardization aids consistency: ISO-15919's role in standardizing transliterations across different Indic scripts helped in creating a more consistent input for the NLU models.
- Topic-specific improvements: Even in high-resource languages like Hindi, where overall improvements were modest, specific topics showed significant gains with alternate representations, indicating their potential for targeted improvements.
Key Results
- Overall improvement in low-resource languages: For Tamil and Marathi, IndicSOUNDEX showed significant improvements, with IndicSOUNDEX outperforming ISO-15919 in some cases. For Telugu, both representations showed improvements in a subset of topics.
- Hindi results were mixed: For Hindi, neither IndicSOUNDEX nor ISO-15919 showed an overall statistically significant improvement. However, specific topics like SmartHome and Global saw improvements with both methods. ISO-15919 even showed a statistically significant overall degradation for Hindi.
- Quantifiable improvements: For Tamil, IndicSOUNDEX led to an average SemER drop of 11% across improved topics, while ISO-15919 achieved a 13% drop. For Marathi, IndicSOUNDEX showed an 18% drop in improved topics, and ISO-15919 showed a 10% drop.
- Table 6 summarizes percentage changes: IndicSOUNDEX resulted in a -0.07% change for Hindi and -7.6% for Tamil, while ISO-15919 showed +1.08% for Hindi and -6.2% for Tamil.
Limitations and Future Work
- Limitations:
- The paper notes that for high-resource languages like Hindi, the impact of alternate representations was less pronounced. This is attributed to the high consistency of transcription and the availability of sufficient training data, which already allows models to learn variations.
- The impact of IndicSOUNDEX for Telugu warrants further investigation.
- Future Work:
- Exploring more advanced model architectures, such as transformer models.
- Investigating pre-training models with a combination of original tokens and alternate representations.
- Exploring the use of character bigrams (instead of unigrams) in embedding layers, considering both original text and alternate representations.
Summary updated