Abstract
Neural Machine Translation (NMT) models often rely on sub-word segmentation to manage large vocabularies, yet the impact of these choices on the model's internal linguistic knowledge remains under-explored. This paper investigates how different input granularities—words, Byte-Pair Encoding (BPE), Morfessor, and characters—affect the quality of learned representations. Through probing classifiers, we demonstrate that while character-based models are superior for morphology and noise resistance, subword units better capture syntactic dependencies. Our results suggest that no single granularity is optimal for all linguistic tasks, highlighting a fundamental trade-off in NMT representation learning.
Introduction
The success of contextualized word representations in downstream NLP tasks has led to increased interest in the internal states of pre-trained models, including those trained for NMT. However, NMT models face a significant "vocabulary problem" due to memory constraints, necessitating the use of sub-word units like BPE or individual characters. While the impact of these choices on translation quality (BLEU) is well-documented, their effect on the model's acquisition of morphological, syntactic, and semantic knowledge is less understood. This study addresses this gap by probing the hidden states of NMT encoders across various granularities.
Methodology
The researchers employed a probing (diagnostic) classifier framework to evaluate the representations learned by encoder-decoder LSTM models.
NMT Model Training
Models were trained on standard benchmarks (WMT, IWSLT) for four language pairs: German-English, Czech-English, Russian-English, and English-German. Four input granularities were compared:
- Word-based: Standard whole-word units.
- BPE: Frequency-based subword units.
- Morfessor: Linguistically-motivated subword units.
- Character-based: Individual characters.
Probing Framework
Hidden states were extracted from the trained encoders. For words split into multiple units (BPE or characters), two aggregation strategies were tested: Average (mean of all units) and Last (final unit activation). Simple logistic regression classifiers were then trained to predict linguistic tags:
- Morphology: Part-of-Speech (POS) tagging.
- Semantics: Lexical semantic tagging.
- Syntax: Combinatory Categorial Grammar (CCG) supertagging.
Additionally, the models' robustness was tested by introducing synthetic noise (swapped letters) and real-world typos into the test data.
Results and Discussion
The Morphology-Syntax Trade-off
The choice of granularity significantly impacts the type of linguistic information captured. Character-based models excel at morphology because they directly process the internal structure of words (roots, affixes). This is particularly evident in morphologically rich languages like Czech and Russian.
Conversely, character-based models struggle with syntax. Segmenting sentences into characters creates much longer sequences, making it difficult for LSTMs to maintain long-distance grammatical dependencies. Subword units (BPE and Morfessor) provide a better balance, outperforming characters in syntactic tasks by approximately 1-2%.
Representation Aggregation and Linguistic Motivation
A key finding was that the Last activation of a multi-unit word consistently outperformed the Average activation. This suggests that the model's internal state accumulates information sequentially, with the final unit serving as a summary of the entire word. Furthermore, linguistically-motivated segmentation (Morfessor) proved more effective than frequency-based segmentation (BPE) for morphological tasks in complex languages.
Robustness to Noise
Character-based representations demonstrate remarkable resilience to input noise. While word and BPE-based models suffer significant performance degradation when faced with typos—often resulting in "Unknown" tokens—character models maintain high accuracy.
As shown in Figure 6, character-based models exhibit a much shallower decline in BLEU scores as noise increases compared to word-based or subword-based models. This robustness is attributed to the fact that surrounding characters often provide sufficient context to recover meaning even when individual characters are corrupted.
Conclusion
This research demonstrates that the choice of input granularity in NMT is not merely a technical necessity but a decision that shapes the model's linguistic understanding. Character-based models are the champions of morphology and robustness, whereas subword units are better suited for capturing complex syntax. The ultimate performance gain observed by concatenating different representations (Word + BPE + Character) suggests that these granularities provide complementary information, pointing toward hybrid architectures as a promising direction for future NMT research.
Note: This study focused on LSTM architectures; future work is needed to determine if these findings generalize to Transformer-based models.
Summary updated