Introduction
Neural Machine Translation (NMT) has largely superseded Statistical Machine Translation (SMT) due to its superior fluency and end-to-end trainability. However, NMT models, typically based on the Transformer architecture, often function as "black boxes" lacking the explicit word alignment inherent in SMT. While the attention mechanism provides a latent form of alignment, it is often linguistically inaccurate. Recent research presented at IJCAI-22, "Explicit Alignment Learning for Neural Machine Translation," addresses this by introducing a self-training method to induce explicit alignment without requiring external tools or additional parameters.
Methodology: Explicit Alignment Learning (EAL)
The proposed Explicit Alignment Learning (EAL) framework aims to align source and target word embeddings within the shared latent space of the Transformer. The process consists of two primary phases:
- Warmup Phase: The model undergoes standard NMT training to ensure the cross-attention mechanism begins to produce meaningful weights.
- Self-Training with Embedding Mixup: In this phase, the model utilizes its own cross-attention weights to identify correspondences between source and target tokens. A "sampling and replacement" strategy is employed where a portion of source word embeddings are replaced with the embeddings of their aligned target words.
By injecting target embeddings into the source input, the model is forced to treat them as semantically interchangeable in context. This "denoising" task drives the source and target embeddings into a unified spatial context, facilitating better alignment and translation. Crucially, EAL reuses the existing Transformer structure, adding zero new parameters.
Experimental Setup
The effectiveness of EAL was evaluated across several benchmarks:
- High-Resource: WMT14 English-to-German (En→De), English-to-French (En→Fr), and WMT17 Chinese-to-English (Zh→En).
- Low-Resource: IWSLT14 German-to-English (De→En) and IWSLT13 French-to-English (Fr→En).
Results and Analysis
EAL demonstrated consistent improvements in translation quality across all tested language pairs. As shown in the results table, the method provided significant BLEU gains over baseline Transformer models.
Key findings include:
- Translation Quality: EAL improved the Transformer-base model by 1.01 BLEU on WMT14 En→De and up to 1.46 BLEU on low-resource tasks.
- Alignment Accuracy: The Alignment Error Rate (AER) was significantly reduced. While standard Transformers often exhibit an AER around 66%, EAL reduced this to 32.0%, rivaling models trained with external SMT supervision (e.g., GIZA++).
- Layer Sensitivity: Analysis revealed that middle decoder layers (specifically Layer 4 in a 6-layer setup) provide the most accurate linguistic alignment, as top layers become overly specialized for final token prediction.
Conclusion
Explicit Alignment Learning offers a robust and efficient path toward more interpretable and higher-performing NMT. By enabling models to "teach themselves" alignment through embedding mixup, EAL bridges the gap between the transparency of SMT and the power of NMT. Future work may extend this approach to multilingual settings and explore more sophisticated token replacement strategies.
Summary updated