Enhanced Training Methods for Multiple Languages

Hai Li, Yang Li2023 · DOI

Abstract

Document-grounded dialogue generation based on multilingual is a challenging and realistic task. Unlike previous tasks, it need to tackle with multiple high-resource languages facilitating low-resource languages. This paper summarizes our research based on a three-stage pipeline that includes retrieval, re-rank and generation where each component is individually optimized. In different languages with limited data scenarios, we mainly improve the robustness of the pipeline through data augmentation and embedding perturbation with purpose of improving the performance designing three training methods: cross-language enhancement training, weighted training with neighborhood distribution augmentation, and ensemble adversarial training, all of that can be used as plug and play modules. Through experiments with different settings, it has been shown that our methods can effectively improve the generalization performance of pipeline with score ranking 6th among the public submissions on leaderboards.

Introduction

Document-grounded dialogue systems aim to provide precise answers to user questions by searching through specific documents. While these systems perform reliably in high-resource languages like English and Chinese, they often struggle in low-resource languages such as Vietnamese or French, where training data is scarce.

The challenge is two-fold: first, effectively transferring knowledge from data-rich languages to data-poor ones; and second, ensuring model robustness against the semantic variations inherent in natural conversation. This research addresses these issues within a three-stage pipeline: Retrieval, Re-ranking, and Generation.

Methodology

To enhance the performance of multilingual dialogue systems, we propose three modular, "plug-and-play" training enhancements that can be integrated into existing architectures.

Overview of the three-stage pipeline and the integration of the proposed plug-and-play modules.

1. Cross-Language Enhancement Training (CET)

CET utilizes a token exchange strategy based on word frequency. By mapping tokens from high-resource languages to low-resource ones according to their frequency rank, the model is forced to learn shared embeddings across languages. This leverages the insight that high-frequency words across different languages often share similar semantic roles.

2. Enhanced Weighted Training of Neighborhood Distribution (EWTND)

This data augmentation technique identifies frequent words and finds their "semantic neighbors" using vector retrieval. During training, words are randomly replaced by these neighbors. The loss function is then weighted by the similarity of the replacement to ensure the model learns semantic variety without sacrificing accuracy.

3. Ensemble Adversarial Training (EAT)

To improve robustness, EAT combines two adversarial methods: the Fast Gradient Method (FGM) and Projected Gradient Descent (PGD). These methods introduce calculated perturbations into word embeddings during training, teaching the model to ignore noise and reduce generation bias.

Experimental Results

The study utilized the Doc2dial dataset, focusing on English and Chinese (high-resource) and French and Vietnamese (low-resource). The pipeline was evaluated using F1, SacreBleu, and Rouge-L metrics.

Ablation study showing the performance impact of CET, EWTND, and EAT on the Retriever and mT5 generator.

As shown in the results, a standard mT5 model achieved a baseline score of 173.42. By integrating all three proposed methods (CET + EWTND + EAT), the score improved significantly to 188.62. EWTND provided the most substantial individual gain among the augmentation methods. This combined approach allowed the pipeline to rank 6th among public submissions on the official Doc2dial leaderboard.

Discussion and Conclusion

The proposed methods demonstrate that controlled perturbations—whether at the semantic level (EWTND) or the mathematical level (EAT)—lead to better generalization on unseen data. However, these enhancements come with trade-offs. Ensemble Adversarial Training increases training time by approximately 1.5 times. Furthermore, the EWTND method requires careful hyperparameter tuning; if the number of neighbors or replacement probability is too high, the model may suffer from semantic drift.

Future work will explore more complex sampling methods for adversarial training and extend cross-language mapping to a broader range of diverse language pairs.

Summary updated

Abstract

Document-grounded dialogue generation based on multilingual is a challenging and realistic task. Unlike previous tasks, it need to tackle with multiple high-resource languages facilitating low-resource languages. This paper summarizes our research based on a three-stage pipeline that includes retrieval, re-rank and generation where each component is individually optimized. In different languages with limited data scenarios, we mainly improve the robustness of the pipeline through data augmentation and embedding perturbation with purpose of improving the performance designing three training methods: cross-language enhancement training, weighted training with neighborhood distribution augmentation, and ensemble adversarial training, all of that can be used as plug and play modules. Through experiments with different settings, it has been shown that our methods can effectively improve the generalization performance of pipeline with score ranking 6th among the public submissions on leaderboards.
BibTeX
@misc{li2023enhanced,
  author = {Hai Li and Yang Li},
  title = {{Enhanced Training Methods for Multiple Languages}},
  year = {2023},
  doi = {10.18653/v1/2023.dialdoc-1.6},
}
Figure