Abstract
This study investigates the relationship between Neural Machine Translation (NMT) quality and the semantic richness of learned sentence representations. By introducing a "compound attention" architecture that preserves a fixed-size sentence bottleneck, the authors demonstrate a surprising inverse correlation: models with higher BLEU scores often produce inferior embeddings for downstream semantic and classification tasks. The findings suggest that optimizing for n-gram overlap metrics like BLEU may encourage models to prioritize surface-level patterns over deep linguistic meaning.
Introduction
In the early development of deep learning for NLP, NMT encoders were frequently used to generate "universal" sentence embeddings—fixed-length vectors intended to capture the core meaning of a sentence. However, the introduction of the attention mechanism significantly shifted the landscape. By allowing decoders to attend to any part of the source sequence dynamically, attention removed the necessity for a single "bottleneck" vector, leading to massive gains in translation quality (BLEU scores) but complicating the extraction of static representations.
This research addresses two fundamental questions: Can we design an NMT architecture that utilizes attention while still producing a fixed-size sentence representation? And does a higher BLEU score necessarily imply a better representation of meaning?
Methodology: Compound Attention
To bridge the gap between fully attentive models and fixed-size representations, the authors propose a novel Compound Attention architecture. This model restores a bottleneck while maintaining the performance benefits of attention.
As shown in Figure 1, the architecture consists of:
- Encoder with Inner Attention: Instead of relying solely on the final hidden state, the encoder uses multiple attention heads to compute a weighted sum of all encoder states. This produces a structured matrix $M$ of fixed size, which serves as the sentence embedding.
- Attentive Decoder: The decoder performs a second round of attention, but instead of attending to the raw sequence of words, it attends to the rows of the matrix $M$.
This approach was compared against several baselines, including standard RNN "FINAL" states, mean/max pooling, and modified Transformers with inner-attention bottlenecks.
Experimental Setup
The models were trained on two translation pairs: English-to-German (using the Multi30k dataset) and English-to-Czech (using the 57M-pair CzEng 1.7 corpus). Evaluation was conducted using SentEval, a suite of 10 classification tasks (e.g., sentiment, subjectivity) and 7 semantic similarity tasks (STS). Additionally, paraphrase tasks using COCO and HyTER were employed to assess how well equivalent sentences clustered in the embedding space.
The BLEU-Meaning Trade-off
The most striking finding of the study is the clear negative correlation between translation performance and representation quality. In the Czech models, the correlation between BLEU and average classification accuracy was found to be -0.57.
As illustrated in Figure 3, increasing the complexity of the attention mechanism (e.g., adding more heads) generally improved BLEU scores but consistently degraded the quality of the learned sentence representations. The authors hypothesize that NMT models trained with standard cross-entropy loss become "lazy"; to maximize BLEU, they focus on matching surface-level word forms and positions rather than deriving deep semantic structures.
Furthermore, visualization of the attention heads revealed that they often failed to learn semantic roles (like "agent" or "patient"). Instead, they tended to divide the sentence into equal positional segments, making the representations brittle to syntactic changes like passivization (e.g., "The dog bit the man" vs. "The man was bitten by the dog").
Conclusion
The research highlights a fundamental tension in current NMT paradigms: the metrics we use to measure translation success may be at odds with the goal of learning universal meaning. While Compound Attention models perform nearly as well as fully attentive models in translation, they are often outperformed by simpler models—or even basic word averaging—on semantic benchmarks.
The authors suggest that the reliance on single-reference BLEU may be the root cause, as it rewards exact n-gram overlaps over semantic equivalence. Future work may involve exploring loss functions that penalize semantic deviations or investigating whether multilingual NMT can force models to develop more robust, language-independent representations.
Summary updated