Divide and Rule: Training Context-Aware Multi-Encoder Translation Models with Little Resources

Lorenzo Lupo, Marco Dinarelli, Laurent Besacier

Introduction

Standard Neural Machine Translation (NMT) systems typically operate at the sentence level, often failing to resolve ambiguities that require broader document context. This leads to errors in discourse-heavy phenomena, such as pronoun gender agreement or ellipsis resolution. While context-aware models exist, they are notoriously difficult to train due to the "double challenge of sparsity": the rarity of tokens requiring context (signal sparsity) and the difficulty of finding relevant clues in long sequences (context sparsity).

The "Divide and Rule" Strategy

To address these challenges, researchers proposed a "divide and rule" (d&r) pre-training strategy. The core idea is to artificially increase the density of the contextual signal by splitting training sentences into shorter segments. By breaking internal syntactic and semantic relations, the model is forced to rely on the "context" (the other segment or previous sentences) to reconstruct the original meaning.

The multi-encoder architecture utilizes a dedicated context encoder and integration module to process surrounding information.

The architecture involves a multi-encoder setup where a primary encoder processes the source sentence, while a secondary context encoder processes surrounding segments. These representations are then merged via a context integration module before being passed to the decoder. The authors explored several splitting variants, including a simple "Middle-split" (cutting sentences in half) and more complex linguistic methods like "Synt-split" which targets coreference relations.

Experimental Results

The d&r strategy was evaluated on English→German (En→De), English→French (En→Fr), and English→Russian (En→Ru) translation tasks. Beyond standard BLEU scores, the researchers used contrastive test sets like ContraPro (pronoun accuracy) and EllipsisVP (verb phrase ellipsis) to measure discourse-level performance.

Main results showing significant improvements in discourse-aware metrics (ContraPro and Ellipsis-VP) using the d&r strategy, especially in low-resource settings.

As shown in the results table, d&r models significantly outperform standard multi-encoder baselines. In the En→De task, the K1-d&r model improved pronoun accuracy from 47.05% to 60.21% in low-resource settings. Similar massive gains were observed in Russian ellipsis resolution, where accuracy jumped from 32.20% to 47.00%. Notably, these improvements occurred even when BLEU scores remained relatively flat, suggesting that context-aware gains are often hidden from standard metrics.

Discussion and Conclusion

The d&r approach not only improves accuracy but also enhances data efficiency and training speed. Models trained with d&r on a fraction of the data often outperformed those trained on full datasets without the strategy. This makes context-aware NMT more feasible for low-resource languages and computationally constrained environments. Surprisingly, the simple "Middle-split" method proved robust across languages, suggesting the strategy is easily applicable without heavy NLP preprocessing.

Summary updated

BibTeX
@inproceedings{lupo2022divide,
  author = {Lorenzo Lupo and Marco Dinarelli and Laurent Besacier},
  title = {{Divide and Rule: Training Context-Aware Multi-Encoder Translation Models with Little Resources}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.312},
}
Figure