Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation

Zhengxin Yang, Jinchao Zhang, Fandong Meng, Shuhao Gu, Yang Feng, Jie Zhou

Abstract

Context modeling is essential to generate coherent and consistent translation for Document-level Neural Machine Translations. The widely used method for document-level translation usually compresses the context information into a representation via hierarchical attention networks. However, this method neither considers the relationship between context words nor distinguishes the roles of context words. To address this problem, we propose a query-guided capsule networks to cluster context information into different perspectives from which the target translation may concern. Experiment results show that our method can significantly outperform strong baselines on multiple data sets of different domains.

Introduction

Translating a document requires more than just processing isolated sentences. A robust model must maintain consistency in tenses, resolve anaphoric pronouns, and ensure uniform noun expressions throughout the text. While sentence-level Neural Machine Translation (NMT) has achieved high performance, document-level translation remains a significant challenge due to the complexity of capturing long-range dependencies and contextual nuances.

Problem Statement

Most existing document-level NMT models rely on Hierarchical Attention Networks (HAN) to incorporate context. However, HAN suffers from two primary limitations:

  1. Word Isolation: It calculates the relevance of each context word to the current word independently, failing to model how context words relate to one another.
  2. Role Ambiguity: It struggles to distinguish between the multiple roles a single context word might play across different linguistic "perspectives," such as tense consistency versus entity resolution.

Proposed Method: Query-Guided Capsule Networks

To overcome these limitations, we propose a novel architecture integrated into the standard Transformer model that utilizes Query-Guided Capsule Networks (QCN) to cluster context information into distinct perspectives.

Overall architecture of the proposed document-level NMT model, featuring the QCN and the sub-layer-expanded Transformer.

Query-Guided Capsule Network (QCN)

The QCN is the core innovation of this work. Unlike standard capsule networks that extract features in an unsupervised manner, QCN uses the current source sentence as a "query" to guide feature extraction from historical sentences. This ensures that the resulting high-level capsules represent information specifically relevant to the current translation task.

Detailed mechanism of the Query-Guided Capsule Network (QCN) showing the dynamic routing process guided by the query vector.

We modified the standard dynamic routing algorithm to incorporate Pearson Correlation Coefficients (PCCs). PCCs measure the linear correlation between the query and the input capsules, allowing the routing process to focus on "higher-level" capsules that align with the current sentence's semantic needs.

Sub-layer-expanded Transformer and Regularization

The architecture includes a sub-layer-expanded Transformer encoder that uses multi-head attention to integrate the contextual features extracted by the QCN. Additionally, a regularization layer is employed during training. This layer uses PCCs to minimize the semantic distance between the capsule representations of the source and target sentences, ensuring they inhabit a shared semantic space.

Experimental Results

The model was evaluated on English-to-German translation tasks across three domains: TED (informal/spoken), News (WMT’16), and Europarl (formal proceedings).

Our QCN-enhanced model consistently outperformed both the standard Transformer and strong document-level baselines:

  • TED Dataset: Achieved a gain of +0.59 BLEU and +0.61 Meteor over previous state-of-the-art context-aware models.
  • Europarl: Reached a BLEU score of 29.82, a significant improvement over the base Transformer's 28.72.

Qualitative analysis through heatmap visualizations of coupling coefficients confirmed that the model successfully learns to "vote" for specific feature capsules representing different contextual perspectives, validating the effectiveness of the guided routing mechanism.

Conclusion

By modeling context as a "part-whole" relationship through Query-Guided Capsule Networks, our approach effectively clusters linguistic information into useful perspectives for translation. The introduction of query-guided routing and PCC-based regularization allows for a more nuanced and relevant integration of document-level context, leading to superior translation quality across diverse domains.

Summary updated

Abstract

Context modeling is essential to generate coherent and consistent translation for Document-level Neural Machine Translations. The widely used method for document-level translation usually compresses the context information into a representation via hierarchical attention networks. However, this method neither considers the relationship between context words nor distinguishes the roles of context words. To address this problem, we propose a query-guided capsule networks to cluster context information into different perspectives from which the target translation may concern. Experiment results show that our method can significantly outperform strong baselines on multiple data sets of different domains.
BibTeX
@inproceedings{yang2019enhancing,
  author = {Zhengxin Yang and Jinchao Zhang and Fandong Meng and Shuhao Gu and Yang Feng and Jie Zhou},
  title = {{Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=94IvD2RXHU},
}
Figure