Introduction
Standard Neural Machine Translation (NMT) systems, such as the Transformer, typically process sentences in isolation. This "context-blindness" leads to significant errors in translating pronouns (e.g., "it," "they") or handling elliptical expressions where information is omitted based on previous discourse. For instance, in movie subtitles, a pronoun's gender or plurality often depends on a noun mentioned several sentences prior.
While previous research has attempted to incorporate context by concatenating multiple sentences, these methods often suffer from quadratic computational complexity and "long-range confusion," where the model struggles to identify relevant signals within excessively long input sequences.
The Hierarchical Context Encoder (HCE)
To address these limitations, this paper introduces the Hierarchical Context Encoder (HCE). Unlike flat concatenation models, HCE employs a multi-level strategy to digest surrounding context efficiently.
The HCE architecture consists of four primary stages:
- Sentence-Level Encoding: Each preceding context sentence is processed individually through a Fully Attentional Network (FAN).
- Self-Attentive Embedding: To maintain efficiency, an "attentive weighted sum" compresses the variable-length encoded sentences into fixed-size sentence embeddings.
- Context-Level Encoding: These embeddings are fed into a secondary encoder that captures inter-sentence relationships, allowing the model to understand the broader narrative flow without word-level overhead.
- Gated Integration: A gated sum mechanism dynamically determines the relevance of the context to the current source sentence, preventing irrelevant information from degrading translation quality.
Experimental Results
The HCE model was evaluated against several state-of-the-art context-aware baselines, including Deep Context-to-Context (DCT), Document-Aware Transformer (DAT), and Hierarchical Attention Network (HAN). Testing spanned multiple datasets: IWSLT’17 (TED talks), OpenSubtitles (En-De, En-Tr), and a custom web-crawled English-Korean subtitle corpus.
Performance and Efficiency
HCE consistently outperformed all baselines in terms of BLEU scores. The improvement was most pronounced in the English-Korean task, where HCE achieved a score of 11.30, compared to the context-less Transformer's 8.58.
Beyond accuracy, HCE proved to be the most efficient architecture. It recorded a training speed of 4.67 steps/sec and an inference time of 65.12 tokens/sec, surpassing other hierarchical models. This efficiency stems from the model's ability to cache context representations, making it highly suitable for real-time applications.
Pronoun Resolution and Context Utility
The authors developed a specialized pronoun resolution test suite for English-Korean. HCE achieved 48% accuracy, nearly doubling the performance of the standard Transformer (25%). Furthermore, in a "helpful/unhelpful" test set—where context is strictly required for correct translation—HCE was the only model to significantly close the performance gap, demonstrating a genuine ability to leverage surrounding information.
Conclusion
The Hierarchical Context Encoder provides a scalable and effective solution for context-aware NMT. By utilizing self-attentive sentence embeddings and a gated integration mechanism, it achieves superior translation quality and inference speed. Future work may explore incorporating target-side context and expanding specialized benchmarks to further validate the model's reasoning capabilities in diverse linguistic scenarios.
Summary updated