TL;DR
This paper tackles the quadratic computational cost of Transformer's attention mechanism, which hinders its ability to process long sequences like entire documents for machine translation. They adapt a Locality-Sensitive Hashing (LSH) technique to make attention more efficient, enabling larger context windows for document-level translation, though with a slight trade-off in translation quality.
What Problem Does This Paper Address?
The Transformer architecture, a cornerstone of modern Natural Language Processing (NLP), relies heavily on the self-attention mechanism. While this mechanism allows for parallel processing and captures long-range dependencies, its computational complexity scales quadratically with the input sequence length ($O(L^2)$). This becomes a significant bottleneck when dealing with long texts, such as entire documents, where the sequence length can be substantial. Existing approaches to mitigate this often involve limiting the context to a few sentences or compressing document information, which may not fully leverage the available context. This paper aims to enable Transformers to effectively process much longer sequences for tasks like document-level Neural Machine Translation (NMT) without prohibitive computational costs.
Methods
The core of this work is the adaptation of Locality-Sensitive Hashing (LSH) for attention mechanisms in NMT. LSH is a technique that hashes similar items into the same "buckets" with high probability, allowing for efficient approximate nearest neighbor search.
The authors extend the LSH attention approach proposed by Kitaev et al. (2020), which was initially designed for language modeling. Their contributions include:
- LSH for Self-Attention: They apply LSH to the self-attention mechanism within the Transformer NMT framework. This involves sorting keys and queries by their hash values and then processing attention within smaller "chunks" of these sorted sequences, rather than the entire sequence.
- LSH for Cross-Attention: They extend the LSH concept to cross-attention, which is crucial for NMT where queries (from the decoder) attend to keys and values (from the encoder). This requires aligning query chunks with corresponding key chunks, a novel aspect of their work.
- Memory Efficiency: By restricting attention to these smaller, hashed chunks, the computational complexity is reduced, making it feasible to process much longer sequences.
- Multi-round LSH: They also explore multi-round hashing, using multiple hash functions to improve the robustness and coverage of the LSH approximation.
Datasets
The experiments were conducted on several standard machine translation datasets:
- WMT 2016 Romanian-English: 612k parallel sentences.
- WMT 2019 English-German: 329k parallel sentences.
- IWSLT 2017 English-German: 232k parallel sentences.
- IWSLT 2017 English-Italian: 206k parallel sentences.
For document-level experiments, the WMT EN-DE and IWSLT EN-DE/EN-IT datasets were grouped by document. They also used the ContraPro dataset for evaluating pronoun resolution accuracy in document-level NMT.
Key Insights and Ideas
- LSH is a viable, albeit approximate, replacement for full attention: The experiments show that LSH attention can be used in NMT, but it comes at the cost of a slight degradation in translation quality.
- Document-level NMT benefits most from LSH: The quadratic complexity of full attention becomes a severe limitation for document-level tasks. LSH effectively overcomes this by enabling significantly larger context windows that would be impossible with standard Transformers.
- Self-attention is more robust to LSH than cross-attention: Replacing self-attention with LSH resulted in smaller performance drops compared to replacing cross-attention. This suggests that the decoder's reliance on precise encoder information is more sensitive to approximations.
- Larger context doesn't always mean better translation quality: While LSH allows for much larger context windows in document-level NMT, simply including more context did not consistently improve BLEU/TER scores or performance on targeted test suites like ContraPro. This implies that the model might not be effectively utilizing all the provided context, or that the task itself doesn't always benefit from extremely long-range dependencies.
Key Results
- Sentence-level NMT: LSH-approximated attention performed slightly worse than full attention, with BLEU scores dropping by at most 1% for most systems.
- Document-level NMT:
- LSH enabled training with context sizes up to 12 sentences, which was impossible for full attention systems due to memory constraints.
- Full attention systems showed significant memory savings when using LSH, especially with larger context sizes.
- While LSH allowed for larger context, translation quality (BLEU/TER) did not consistently improve with increased context. However, accuracy on the ContraPro pronoun resolution task significantly improved with context up to three sentences.
- LSH in self-attention alone was more memory-efficient and scaled better with context than full attention.
Limitations and Future Work
- Translation Quality Trade-off: The primary limitation is the slight decrease in translation quality observed with LSH attention, particularly for sentence-level tasks.
- Context Utilization: The paper notes that increasing context size in document-level NMT did not always lead to better translation quality, suggesting that models may not be fully leveraging the extended context.
- Future Work: The authors plan to apply this LSH approach to speech translation, where long input sequences are also a pressing issue. They also suggest further exploration into optimizing the LSH parameters and understanding how different hash functions and rounds affect performance.
Summary updated