One Size Does Not Fit All: Comparing NMT Representations of Different Granularities

Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, Preslav Nakov

Abstract

Neural Machine Translation (NMT) models often rely on sub-word segmentation to manage large vocabularies, yet the impact of these choices on the model's internal linguistic knowledge remains under-explored. This paper investigates how different input granularities—words, Byte-Pair Encoding (BPE), Morfessor, and characters—affect the quality of learned representations. Through probing classifiers, we demonstrate that while character-based models are superior for morphology and noise resistance, subword units better capture syntactic dependencies. Our results suggest that no single granularity is optimal for all linguistic tasks, highlighting a fundamental trade-off in NMT representation learning.

Introduction

The success of contextualized word representations in downstream NLP tasks has led to increased interest in the internal states of pre-trained models, including those trained for NMT. However, NMT models face a significant "vocabulary problem" due to memory constraints, necessitating the use of sub-word units like BPE or individual characters. While the impact of these choices on translation quality (BLEU) is well-documented, their effect on the model's acquisition of morphological, syntactic, and semantic knowledge is less understood. This study addresses this gap by probing the hidden states of NMT encoders across various granularities.

Methodology

The researchers employed a probing (diagnostic) classifier framework to evaluate the representations learned by encoder-decoder LSTM models.

NMT Model Training

Models were trained on standard benchmarks (WMT, IWSLT) for four language pairs: German-English, Czech-English, Russian-English, and English-German. Four input granularities were compared:

  • Word-based: Standard whole-word units.
  • BPE: Frequency-based subword units.
  • Morfessor: Linguistically-motivated subword units.
  • Character-based: Individual characters.

Probing Framework

Hidden states were extracted from the trained encoders. For words split into multiple units (BPE or characters), two aggregation strategies were tested: Average (mean of all units) and Last (final unit activation). Simple logistic regression classifiers were then trained to predict linguistic tags:

  • Morphology: Part-of-Speech (POS) tagging.
  • Semantics: Lexical semantic tagging.
  • Syntax: Combinatory Categorial Grammar (CCG) supertagging.

Additionally, the models' robustness was tested by introducing synthetic noise (swapped letters) and real-world typos into the test data.

Results and Discussion

The Morphology-Syntax Trade-off

The choice of granularity significantly impacts the type of linguistic information captured. Character-based models excel at morphology because they directly process the internal structure of words (roots, affixes). This is particularly evident in morphologically rich languages like Czech and Russian.

Classifier accuracy for POS tagging across different input granularities and language pairs.

Conversely, character-based models struggle with syntax. Segmenting sentences into characters creates much longer sequences, making it difficult for LSTMs to maintain long-distance grammatical dependencies. Subword units (BPE and Morfessor) provide a better balance, outperforming characters in syntactic tasks by approximately 1-2%.

Representation Aggregation and Linguistic Motivation

A key finding was that the Last activation of a multi-unit word consistently outperformed the Average activation. This suggests that the model's internal state accumulates information sequentially, with the final unit serving as a summary of the entire word. Furthermore, linguistically-motivated segmentation (Morfessor) proved more effective than frequency-based segmentation (BPE) for morphological tasks in complex languages.

Robustness to Noise

Character-based representations demonstrate remarkable resilience to input noise. While word and BPE-based models suffer significant performance degradation when faced with typos—often resulting in "Unknown" tokens—character models maintain high accuracy.

Impact of induced noise on translation quality (BLEU) for different granularities.

As shown in Figure 6, character-based models exhibit a much shallower decline in BLEU scores as noise increases compared to word-based or subword-based models. This robustness is attributed to the fact that surrounding characters often provide sufficient context to recover meaning even when individual characters are corrupted.

Conclusion

This research demonstrates that the choice of input granularity in NMT is not merely a technical necessity but a decision that shapes the model's linguistic understanding. Character-based models are the champions of morphology and robustness, whereas subword units are better suited for capturing complex syntax. The ultimate performance gain observed by concatenating different representations (Word + BPE + Character) suggests that these granularities provide complementary information, pointing toward hybrid architectures as a promising direction for future NMT research.

Note: This study focused on LSTM architectures; future work is needed to determine if these findings generalize to Transformer-based models.

Summary updated

BibTeX
@inproceedings{durrani2019one,
  author = {Nadir Durrani and Fahim Dalvi and Hassan Sajjad and Yonatan Belinkov and Preslav Nakov},
  title = {{One Size Does Not Fit All: Comparing NMT Representations of Different Granularities}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1154},
}
Figure