Retraction Note: This paper was published at ACL 2021 but has since been retracted. The authors were found to have reused significant portions of text and ideas from a previously published paper without proper attribution. The summary below describes the scientific content as presented in the original document for informational purposes.
Introduction
Standard Neural Machine Translation (NMT) systems typically operate on isolated sentences, often losing the broader context necessary for accurate pronoun resolution or terminological consistency. While context-aware NMT models exist, they face a significant corpus bottleneck: a lack of large-scale, high-quality document-level parallel corpora. In contrast, sentence-level parallel data and monolingual documents are abundant. This research proposes a "cross-task pre-training" strategy to leverage these unbalanced resources.
Methodology: Cross-Task Pre-training
The core idea is to jointly train a model on two distinct tasks: sentence-level translation and document-level restoration. This approach ensures the model learns both translation mechanics and document-wide structural dependencies.
Model Architecture
The proposed architecture utilizes a dual-encoder setup within a Transformer framework:
- Sentence Encoder: Processes individual source sentences.
- Global Context Encoder: Captures information from surrounding sentences in a document.
To facilitate knowledge transfer, these encoders share self-attention and feed-forward layers. This allows the massive volume of sentence-level parallel data to refine the foundational representations used by the context encoder.
Pre-training Objectives
The model is pre-trained using two simultaneous objectives:
- Sentence-level Translation: Standard supervised learning using parallel sentence pairs.
- Document-level Restoration: A self-supervised task on monolingual documents. This includes Context-Aware Gap Sentence Restoration (CA-GSR), where the model predicts missing sentences, and Context-Aware Masked Sentence Restoration (CA-MSR), a word-level masking task similar to BERT.
Fine-tuning and Evaluation
After pre-training, the model undergoes two-perspective fine-tuning on smaller document-level parallel datasets. It is trained to translate both individual sentences and full documents, making the final model versatile enough for both standalone and context-dependent translation tasks.
Key Results
The cross-task pre-training strategy yielded substantial improvements across multiple benchmarks:
- Translation Quality: On Chinese-English (ZH-EN) tasks, the model achieved a BLEU score of 50.03, significantly outperforming the baseline Transformer (39.64).
- Discourse Accuracy: The model demonstrated superior performance in resolving discourse-level phenomena. For instance, accuracy in handling "deixis" (context-dependent references) improved from approximately 50% to 81.9%.
- Ablation Studies: Results confirmed that both sentence-level parallel data and monolingual document restoration are essential; removing either component led to a marked decrease in performance.
Conclusion
By breaking the corpus bottleneck through cross-task pre-training, this method allows NMT models to effectively utilize massive monolingual and sentence-level resources to improve document-level translation. While the model introduces additional architectural complexity, the gains in both BLEU scores and discourse-level accuracy suggest that joint learning of translation and context restoration is a powerful paradigm for context-aware NMT.
Note: As mentioned, this work was retracted due to originality concerns. Readers should consult the retraction notice for further details.
Summary updated