Translation systems trained on only a few thousand sentence pairs fail in a specific way. Their attention, the mechanism that lets each word look at every other word, spreads thinly over irrelevant positions instead of locking onto the words that matter. The fixes that work today all push the model toward looking only at nearby words. They do this through shrunken models and heavy dropout, through penalties on attention that jumps far, or through fixed windows around each position. These fixes raise overall translation scores. Nobody has checked what they cost on the sentences where a word's meaning depends on something far away, such as a German separable verb whose particle sits at the end of the clause ("kam … zurück" → "came back"). Corpus-level scores like BLEU barely register those errors, because such sentences are a minority and each error touches only a word or two.
This note proposes a Transformer encoder that is local by default and global only at a coarse grain. In the lower layers, a cheap fixed operator (a depthwise convolution, which mixes each position with a few neighbours) handles nearby context. In the upper layers, a few attention heads reach across the whole sentence, but they attend to whole-word summaries rather than to individual subword pieces. The bet is that low-resource attention blurs partly because it has too many fine-grained targets to learn from too little data. Coarse targets should keep long-range reach without the blur. The note also contributes the measurement that makes this testable: a distance-bucketed evaluation of long-range dependencies in simulated low-resource German–English, where the locality fixes can finally be charged for what they give up.
Core Claim
Two lines of evidence are in tension. Work on attention under data scarcity finds that attention becomes "blurry" when the training set shrinks from 4M to 100k pairs. Entropy- and Distance-Regularized Attention Improves Low-Resource Neural Machine Translation fixes this with entropy penalties and a distance penalty that discourages large jumps. The authors report that below about 50k pairs, hard-coded local masks still do better. Optimizing Transformer for Low-Resource Neural Machine Translation shows that capacity reduction and aggressive dropout recover up to 7.3 BLEU. Even so, an optimized recurrent network still wins below roughly 20k pairs, which points to a missing locality bias. On the other side, An Analysis of Encoder Representations in Transformer-Based Machine Translation shows that upper encoder layers carry long-distance semantic and syntactic structure, with some heads exceeding 35% unlabeled attachment against dependency trees. That non-local structure is also what transfers across language pairs.
Our claim is that both are right, at different layers and at different granularities. Locality should be enforced structurally in the lower layers. Long-range reach should be kept in the upper layers by restricting global attention to word-level keys and values. The specific prediction is that, at 5k–30k pairs, this encoder matches the BLEU of the best locality-enforcing baselines and beats them on the long-distance buckets of the targeted evaluation. The advantage should grow as data shrinks. If hard local masks match it on long-distance buckets too, the claim is wrong.
The word-level choice matters for a reason specific to low resource. Araabi and Monz find that small subword vocabularies are optimal under scarcity, and small vocabularies split words into many pieces. That lengthens the sequence attention must search, which is exactly the regime where blur appears. Pooling to words separates the vocabulary choice from the attention span. It also sidesteps a failure reported in Unsupervised Source Hierarchies for Low-Resource Neural Machine Translation: that paper induced hierarchies with no parser, and almost none of its learned merges recombined subwords into actual words. We get the word level for free from the segmentation markers instead of trying to learn it.
Research Program
The model is a standard encoder–decoder Transformer at the reduced, heavily regularized configuration of Araabi and Monz. Only the encoder changes. In the lower half of the layers, self-attention is replaced by a depthwise convolution branch with a small kernel. In the upper half, each layer keeps a convolution branch and adds a global attention branch. In that branch, queries are subword states, while keys and values are word summaries: the mean of the subword states belonging to each word, identified by the subword continuation markers.
This borrows the local/global split of Lite Transformer with Long-Short Range Attention. That paper splits channels between a convolution branch and an attention branch and observes that many attention heads waste capacity on the diagonal. It was designed and tested for compute budgets, however, not data scarcity, and its global branch attends over subwords. We differ in two ways: the split is assigned by depth, and the global branch attends over words.
The content-grouping idea, which collapses many fine-grained positions into fewer meaningful targets to fight attention dilution, is motivated by ATD: Improved Transformer With Adaptive Token Dictionary for Image Restoration. ATD groups image features into a learned dictionary of categories. We use the cheapest linguistically meaningful grouping available, with nothing learned. The decoder and its attention over the encoder are left untouched so that gains can be attributed to the encoder. A secondary variant also exposes word-level summaries to the decoder's attention over the source.
The analysis side builds the long-range evaluation. On the German side of the IWSLT14 test set, a German dependency parser (available because the low-resource setting is simulated) marks sentences containing a separable-verb particle, an auxiliary–participle pair, or a verb-final subordinate clause. Each sentence is bucketed by the distance in words between the two linked elements. For each bucket we report BLEU and a targeted accuracy: whether the reference's verb lemma appears in the output. This is the instrument that makes the cost of locality visible.
Baselines
The naive reference point is the default Transformer-base trained on each subset. Its main use is to calibrate how much any tuning buys. The strongest tuned reference is the optimized low-resource Transformer of Optimizing Transformer for Low-Resource Neural Machine Translation. On the 5k-pair IWSLT14 De-En subset, its cumulative tuning raised BLEU from 8.3 to 14.3. Its optimized recurrent network is also included, since it wins below roughly 20k pairs. Every proposed model starts from this recipe, and beating it means a gain on top of it at 5k, 10k and 20k pairs rather than a gain over defaults.
The strongest existing attention-side competitor is EaDRA from Entropy- and Distance-Regularized Attention Improves Low-Resource Neural Machine Translation. It reports 13.5 → 16.2 BLEU on a 100k En-De subset. It is also run with the hard local masks the same authors found stronger below 50k pairs. These are the direct rivals, because they attack the same blur by enforcing locality. Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecture, which feeds each layer's query–key projections into the next layer at no parameter cost, is included as the strongest architectural competitor. It averaged +1.0 BLEU at 10k pairs and reached 37.9 BLEU on full IWSLT14 De-En when combined with cutoff augmentation. Random sentence concatenation from Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution, worth about +1.0 BLEU, is included both as a competitor and as an add-on, since it is complementary.
Two ablations isolate the key ingredient. The first is the same depth-split encoder with global attention over subwords instead of words, which is Lite Transformer's split assigned by depth. The second is the convolution-only encoder with no global branch. All systems are parameter-matched within 5%. Each gets the same budget of tuning trials, starting from the Araabi–Monz configuration, and uses the same BPE sweep, the same data and three seeds. Beating a baseline means a seed-robust BLEU gain at 5k–30k pairs that is at least neutral in short-distance buckets. The central result is a gain in the long-distance buckets over EaDRA and hard masks.
Why not just…?
Why not just use an optimized recurrent network below 20k pairs? Araabi and Monz show it wins there, so it is a real baseline rather than a straw man. It also offers no path to the larger-data regimes where Transformers dominate, and no evidence in the file says it handles distant dependencies well. Our distance buckets will show directly whether its sequential bias resolves verb–particle links at long range. If it does and our model does not, the conclusion is that recurrence, not word-level reach, is the missing ingredient.
Why not just use hard local masks or the EaDRA distance penalty? This is the main threat. EaDRA's own authors report that fixed masks are strongest below 50k pairs, and on BLEU alone they may tie with our model. The penalty, however, explicitly punishes long jumps, and masks forbid them outright. The distance-bucketed evaluation settles the question. If masks match us in the longest buckets, word-level global heads add nothing and we would report that.
Why not just use Lite Transformer's convolution-plus-attention split as is? It already combines local and global branches. The subword-key ablation is exactly this model with the split assigned by depth. If it equals the word-key version, our granularity argument fails and the contribution reduces to applying Lite Transformer under data scarcity.
Why not just concatenate random sentences? It is cheap and worth about +1.0 BLEU. Its own authors show that the gain comes from diversity in position, length and context, not from any modeling of dependencies. We expect it to stack with our encoder rather than replace it, and we test the combination.
Proposed Experiments
The main grid covers IWSLT14 De-En subsets of 5k, 10k, 20k, 30k, 100k and the full 165k pairs, plus Galician–English (10k) and Belarusian–English (4.5k) to check transfer beyond German. Metrics are sacreBLEU and chrF over three seeds. On the De-En subsets we add the distance-bucketed long-range evaluation.
Three outcomes would be decisive. The first is whether the word-level encoder's long-distance accuracy exceeds that of EaDRA and hard masks while matching their overall BLEU. The second is whether the gap over the subword-key ablation widens as data shrinks and as the BPE vocabulary gets smaller, which the granularity argument predicts. The third is whether the crossover with the optimized recurrent network moves below 20k pairs.
A supporting analysis measures attention entropy per layer, following EaDRA's diagnostic, to confirm that word-level heads are sharper than subword heads at equal data. It also repeats the attachment-score probing of An Analysis of Encoder Representations in Transformer-Based Machine Translation to test whether the upper-layer global heads recover dependency structure that locality-enforcing baselines lose.
References
- Entropy- and Distance-Regularized Attention Improves Low-Resource Neural Machine Translation
- Optimizing Transformer for Low-Resource Neural Machine Translation
- An Analysis of Encoder Representations in Transformer-Based Machine Translation
- Unsupervised Source Hierarchies for Low-Resource Neural Machine Translation
- Lite Transformer with Long-Short Range Attention
- ATD: Improved Transformer With Adaptive Token Dictionary for Image Restoration
- Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecture
- Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution