Introduction
Neural Machine Translation (NMT) has reached impressive heights for high-resource language pairs. However, it continues to struggle with "low-resource" pairs like English-Hindi, particularly when moving from general conversation to specialized technical domains. Technical fields such as Artificial Intelligence (AI) and Chemistry utilize highly specific vocabularies (e.g., "triacylglycerol" or "hidden layer") that rarely appear in general datasets. This paper explores effective strategies to adapt general-purpose translation models to these niche fields when only a small amount of specialized data is available.
Data and Preprocessing
The researchers utilized the IIT Bombay (IITB) English-Hindi Parallel Corpus (approx. 1.5 million sentence pairs) as the general domain dataset. For technical domains, they used small datasets provided by the AdapMT ICON 2020 shared task.
As shown in Figure 1, the technical datasets for AI and Chemistry are significantly smaller than the general IITB corpus. To handle the morphological richness of Hindi and reduce out-of-vocabulary issues, the team employed SentencePiece tokenization with a vocabulary size of 32,000. Due to hardware constraints, sentences longer than 20 words were filtered from the IITB corpus during training.
Methodology
The study compared two prominent NMT architectures:
- LSTM (Long Short-Term Memory): An encoder-decoder setup with additive attention.
- Transformer: The modern standard for NLP, relying on self-attention mechanisms.
To bridge the domain gap, two primary training strategies were tested:
- Fine-tuning: Training the model on the massive general dataset first, then continuing training on the small technical dataset.
- Mixed-domain Training: Training on a blend of general and technical data simultaneously. To ensure the model did not ignore the small technical datasets, oversampling was used, repeating the technical sentences 10 times.
Results and Discussion
The experimental results demonstrated that mixed-domain training significantly outperformed standard fine-tuning.
As illustrated in Figure 2, the LSTM (mixed-domain) approach was the clear winner, achieving BLEU scores of 16.53 for AI and 9.86 for Chemistry on the development sets. Interestingly, LSTMs outperformed Transformers in this specific task. The authors suggest that Transformers may be more sensitive to hyperparameters and require more data or intensive tuning to excel in very low-resource scenarios.
The researchers also noted the "catastrophic forgetting" trap: simple fine-tuning sometimes performed worse than the general baseline. When the in-domain dataset is tiny, the model can quickly "forget" general language rules. Mixed-domain training successfully mitigated this by maintaining the model's grasp on Hindi grammar from the large corpus while learning specialized terminology. On the final competition test sets, the system achieved BLEU scores of 14.81 for General, 19.08 for AI, and 13.95 for Chemistry domains.
Conclusion
The team's mixed-domain LSTM system ranked 1st in the Chemistry and General domain tasks and 2nd in the AI domain task at the AdapMT ICON 2020 competition. Future work will focus on incorporating longer, more complex sentences and exploring "back-translation" to create synthetic parallel data, further boosting fluency in technical domains.
Summary updated