Mimicking Infants' Bilingual Language Acquisition for Domain Specialized Neural Machine Translation

Chanjun Park, Woo-Young Go, Sugyeong Eo, Hyeonseok Moon, Seolhwa Lee, Heuiseok LimIEEE Access 2022 · DOI · OpenReview

Introduction

Neural Machine Translation (NMT) has achieved remarkable success in general-purpose translation tasks. However, these models often struggle when applied to highly specialized technical domains, such as cybersecurity, medicine, or law. The standard industry practice for addressing this is the Pretrain-Finetuning Approach (PFA), where a model is first trained on a massive general corpus and then specialized using a smaller, domain-specific dataset. While effective to a degree, this sequential method often fails to capture the nuances of technical terminology, leading to mistranslations of critical entities.

Researchers have recently proposed a new training paradigm inspired by cognitive science: the Cross Communication Method (CCM). By mimicking how infants acquire two languages simultaneously, CCM avoids the cognitive biases inherent in sequential learning, offering a superior blueprint for domain-specialized translation.

The Limitations of Sequential Learning

The traditional PFA mimics adult language acquisition. In adults, a well-established first language (L1) creates what cognitive scientists call a "magnet effect." This effect acts as a filter, making it difficult for the brain to map new linguistic patterns in a second language (L2) because the neural pathways are already "set."

In the context of NMT, pretraining on a general corpus (GC) sets the model's weights to favor common language patterns. When the model is later fine-tuned on a specialized corpus (DC), these established weights can interfere with the learning of technical jargon. This "one-sided bias" often results in the model defaulting to general terms even when technical ones are required, a significant hurdle for industrial applications.

The Cross Communication Method (CCM)

To overcome the "magnet effect," the CCM approach draws inspiration from infant bilingualism. Unlike adults, infants do not have a pre-existing linguistic filter; they build neural networks for multiple languages concurrently. CCM applies this principle by training the NMT model on both general and specialized data from scratch.

Comparison between cognitive theories of language acquisition and computational translation models.

The technical implementation of CCM involves several key components:

  • Concurrent Batching: Instead of sequential training, CCM mixes general (GC) and specialized (DC) data into every training batch from the initial step.
  • Corpus Weighting ($\lambda$): A hyperparameter, $\lambda$, controls the ratio of specialized data to general data within each batch, allowing for fine-tuned control over the learning process.
  • Early-Domain-Learning: By learning general syntax and specialized jargon simultaneously, the model avoids the interference seen in PFA, mapping both domains effectively within its internal representations.

Experimental Design

The effectiveness of CCM was tested on a Korean-English translation task within the cybersecurity domain. The researchers utilized a General Corpus (GC) of 2.7 million sentence pairs and a high-quality, human-translated Domain-Specialized Corpus (DC) of 30,000 sentence pairs provided by the National Security Research Institute of South Korea. To test the model's robustness in data-scarce environments, they also simulated an "extremely limited" scenario using only 2,000 specialized pairs.

Results and Discussion

The CCM approach demonstrated significant improvements over both traditional PFA and commercial benchmarks like Google Translate.

Quantitative Performance

On the full cybersecurity dataset, CCM achieved a BLEU score of 91.53, outperforming PFA (90.34) and significantly surpassing Google Translate (55.68). While these scores are high due to the repetitive nature of technical strings in cybersecurity, the relative improvement of CCM is clear.

The Importance of the $\lambda$ Ratio

The study revealed that the optimal ratio of specialized data ($\lambda$) is highly dependent on the total data volume. In scenarios with ample specialized data, a 1:1 ratio (high $\lambda$) yielded the best results. However, in extremely limited data scenarios, a very low $\lambda$ (e.g., 0.01) was necessary to prevent the model from overfitting on the small specialized set while still allowing it to benefit from the unique vocabulary.

Qualitative Fidelity

Human evaluations (on a 0–4 scale) gave CCM a score of 3.94, compared to 2.64 for Google Translate. Qualitative analysis showed that CCM maintained near-perfect fidelity to technical entities. For instance, while other models might mistranslate "DDoS" or omit specific action phrases like "visit websites," CCM consistently produced accurate translations of these domain-specific terms.

Conclusion

The Cross Communication Method represents a shift in how we approach domain specialization in NMT. By moving away from the "adult" sequential learning model and toward an "infant" concurrent learning model, researchers have found a way to mitigate the interference of general language bias. While CCM is more computationally intensive than simple fine-tuning, its superior accuracy in preserving technical entities makes it a compelling choice for high-stakes industrial applications. Future research will focus on scaling this method to more diverse language pairs and fluid domains like creative or legal writing.

Summary updated

Abstract

Existing methods of training domain-specialized neural machine translation (DS-NMT) models are based on the pretrain-finetuning approach (PFA). In this study, we reinterpret existing methods based on the perspective of cognitive science related to cross language speech perception. We propose the cross communication method (CCM), a new DS-NMT training approach. Inspired by the learning method of infants, we perform DS-NMT training by configuring and training DC and GC concurrently in batches. Quantitative and qualitative analysis of our experimental results show that CCM can achieve superior performance compared to the conventional methods. Additionally, we conducted an experiment considering the DS-NMT service to meet industrial demands.
BibTeX
@article{park2022mimicking,
  author = {Chanjun Park and Woo-Young Go and Sugyeong Eo and Hyeonseok Moon and Seolhwa Lee and Heuiseok Lim},
  title = {{Mimicking Infants' Bilingual Language Acquisition for Domain Specialized Neural Machine Translation}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2022.3165572},
}
Figure