TL;DR
This paper investigates extending Byte Pair Encoding (BPE) tokenization to include multi-word expressions (MWEs) for Neural Machine Translation (NMT). Counter-intuitively, a naive approach of adding frequent MWEs actually hurts translation performance. However, using Pointwise Mutual Information (PMI) to select MWEs significantly improves performance, suggesting that the way MWEs are selected is crucial.
What Problem Does This Paper Address?
Traditional subword tokenization methods like BPE break down words into smaller units, which has been highly effective for NMT. However, these methods typically operate within word boundaries. The authors explore whether extending BPE to treat common multi-word expressions (like "New York" or "out of the") as single tokens could further improve NMT. This is motivated by the success of n-grams in older NLP techniques and the intuition that treating fixed phrases as single units might capture semantic meaning more effectively. The core question is: can we improve NMT by allowing BPE to "see" and tokenize across word boundaries, and if so, how?
Methods
The paper proposes two main ways to incorporate MWEs into BPE tokenization for NMT:
- BPE beyond words (Frequency-based): This is the "naive" approach. It extends the standard BPE vocabulary building process to consider frequently occurring contiguous sequences of words (bigrams, trigrams) and even discontinuous sequences (k-skip-n-grams) as potential new tokens. These MWEs are added to the vocabulary based on their raw frequency in the training data.
- PMI-based MWEs: This approach uses Pointwise Mutual Information (PMI) to identify MWEs. PMI is a statistical measure that quantifies the association between words, going beyond simple co-occurrence frequency to find words that are unusually likely to appear together. This method is applied to both contiguous (n-grams) and discontinuous (skip-grams) MWEs. For skip-grams, they introduce a filtering mechanism to ensure the skipped word isn't too ambiguous.
Both methods aim to replace less frequent subword or whole-word tokens in the original BPE vocabulary with these selected MWEs, while keeping the overall vocabulary size constant.
Datasets
The experiments were conducted on two language pairs:
- Hindi-English: Using the IIT Bombay Hindi-English parallel corpus v3.0.
- German-English: Using data from the WMT2019 News Translation task.
The paper notes some filtering of noisy sentence pairs in the German-English training data.
Key Insights and Ideas
- Frequency is not enough: The most surprising finding is that simply adding the most frequent multi-word expressions (like "in_the") as single tokens actually degrades NMT performance. This suggests that common, high-frequency phrases might be too compositionally flexible to be treated as fixed units.
- PMI is key: Using PMI to select MWEs reverses the negative trend and leads to significant improvements in translation quality. This highlights that statistical association, rather than raw frequency, is a better indicator of meaningful multi-word units.
- Incoherent tokens: The paper hypothesizes that frequent MWEs like "in_the" hurt performance because their constituent words ("in" and "the") appear in a vast number of other contexts, making the combined token less coherent and potentially confusing the model.
- Skip-grams can be powerful: Discontinuous MWEs (skip-grams), when selected using PMI, show particular promise, especially when applied to both source and target sides of the translation.
Key Results
- Frequency-based MWEs: Led to a net drop in BLEU and chrF scores across both language pairs compared to the baseline BPE. For example, on the Hi-En dev set, the baseline BLEU was 22.0, while BPE+n/sgms (including frequent skip-grams) dropped to 20.7.
- PMI-based MWEs: Significantly improved performance. For instance, on the De-En dev set, the baseline BLEU was 39.1, while PMI-based N-grams achieved 38.9 (close to baseline), and PMI-based Skip-grams achieved 38.7. However, when looking at the improvement over the baseline, PMI methods generally performed better or on par, with some configurations showing slight gains. Specifically, the "N-grams" (PMI-based) on Hi-En dev reached 22.1, and on De-En dev reached 38.9, often matching or exceeding the baseline.
- Qualitative examples: The paper provides examples where skip-grams correctly capture nuances in translation, but also instances where the model overuses them, leading to incorrect translations.
Limitations and Future Work
- Computational Cost: The authors acknowledge that exploring all combinations of MWEs (e.g., merging bigrams, trigrams, and skip-grams together) would require a significant computational budget for grid search.
- Variable Skip-grams: The current implementation limits skip-grams to word-level skips. Future work could explore character-level or variable-length skip-grams.
- Comparison to Human Annotations: The paper uses automatically extracted MWEs. Future work could compare PMI-based methods to human-annotated MWEs.
- Other NLP Tasks: The study focuses on NMT. Extending these MWE extraction and integration techniques to other NLP tasks is a promising direction.
- Subword Regularization: The paper mentions comparing PMI-based methods to techniques like subword regularization or BPE dropout in future work.
Summary updated