A word aligner decides which word in one sentence translates which word in its translation. The older discriminative aligners did this with a simple rule: compute many weak signals, add their weighted log-scores, and pick the links that score highest. Typical signals were dictionary co-occurrence, distance from the diagonal, part-of-speech match, and spelling similarity. Modern neural aligners instead compute one similarity matrix from a multilingual encoder and either threshold it or pass it through a learned network. We propose bringing back the old additive recipe at the level of whole matrices. Each source of evidence becomes its own "expert" map over all source–target word pairs. Sources include every encoder layer, both directions of a statistical aligner, and string similarity. Each map is normalized on its own, in both alignment directions, and the maps are then combined by a learned weighted sum of log-scores. A cheap element-wise gate lets the spelling expert override semantics for names and numbers.
The claim worth testing is narrow. When only tens to a few hundred human-aligned sentences are available, which is the normal situation outside a few well-funded language pairs, late additive fusion of separately normalized experts should beat two alternatives. One is feeding the same signals as stacked channels into a learned network. The other is adding cross-attention layers. Additive fusion should also leave interpretable per-expert weights and stay symmetric between the two directions by construction. If it fails, we will know whether the decades-old log-linear inductive bias still matters once strong encoders exist.
Where This Sits
Two lines of work meet here.
The first supplies the method. The classical log-linear aligners score an alignment as a weighted sum of scalar feature functions, as in Log-Linear Models for Word Alignment, Improved Discriminative Bilingual Word Alignment and the CRF formulation in Discriminative Word Alignment with Conditional Random Fields. That sum is already additive fusion, but over hand-built scalars, with no continuous representations. We keep their combination rule and replace the features with dense matrices.
The second line motivates the proposal. SimAlign shows that alignment quality peaks at an intermediate mBERT layer (layer 8) and degrades deeper. It then simply picks that one layer. Cross-Align finds its own best extraction point below the top (layer 11). It buys cross-lingual interaction with randomly initialized cross-attention layers that need substantial parallel data. DiscAlign takes hidden states from one stage of a translation model, forms a dot-product similarity matrix, and refines it with a 3×3 convolution trained on gold links. It shows that roughly 1,700 annotated sentences beat statistical aligners trained on millions of lines. None of these combines evidence from several layers or from symbolic priors. All three treat the similarity map as a single object.
Evidence from outside alignment constrains the design. Attention Is Not Always the Answer finds element-wise addition matching or beating concatenation and cross-attention when fusing handcrafted and pretrained speech features. That task is binary frame classification, so it says nothing direct about structured pairwise prediction. DenseSSM shows that shallow details lost in deep layers can be recovered by adding projected shallow states to deep ones. It also shows that raw identity summation underperforms without a selective projection and gate. We take that negative result seriously. Our experts are normalized and weighted before summation, and spelling enters through a learned gate rather than raw addition.
Core Claim
Let $k$ index experts, and let $S^k_{ij}$ be expert $k$'s raw score for linking source word $i$ to target word $j$. The experts are:
- the cosine similarity at each encoder layer, pooled from subwords to words;
- forward and backward statistical-aligner posteriors;
- a normalized edit-distance map;
- a diagonal-distance map.
Each expert is turned into a log-probability by averaging a row softmax and a column softmax, which makes it symmetric under swapping the two languages:
$$\ell^k_{ij} = \tfrac{1}{2}\left(\log \mathrm{softmax}_j(S^k_{i\cdot}/\tau_k) + \log \mathrm{softmax}_i(S^k_{\cdot j}/\tau_k)\right).$$
Here $\tau_k$ is a learned temperature for expert $k$.
The fused link logit is
$$z_{ij} = \sum_k w_k\, \ell^k_{ij} \;+\; g_{ij}\odot \ell^{\text{orth}}_{ij} \;+\; b.$$
The weights $w_k \ge 0$ are learned, $b$ is a bias, and $g_{ij}=\sigma(u^\top[\text{is-number}_i,\text{is-capitalized}_i,\dots])$ is an element-wise gate. The gate is computed from a handful of token-type indicators of the word pair, so string similarity is amplified only where surface form is trustworthy. Links are predicted by thresholding $\sigma(z_{ij})$ and trained with binary cross-entropy on gold sure links. Without the gate there are only about $2K+1$ free parameters, which is why we expect robustness at tiny label budgets.
An optional variant adds DiscAlign's 3×3 convolution as a residual correction on $z$. This tests whether learned structural refinement helps after additive fusion.
One point needs to be stated honestly. A linear layer applied to concatenated channels is mathematically a sum of linear projections. So the claim is not "addition beats concatenation" in the abstract. The specific claims are three:
- Per-expert normalization happens before any mixing.
- Mixing happens in log-probability space, which makes it a product of experts.
- No nonlinearity precedes the sum.
These properties are what we expect to buy sample efficiency and symmetry.
Research Program
We build the fused aligner on top of two backbones. The first is frozen mBERT or XLM-R, which tests whether fusion rescues raw encoders. The second is an AWESOME-fine-tuned mBERT (Word Alignment by Fine-tuning Embeddings on Parallel Corpora), which tests whether gains survive once the encoder is already alignment-tuned.
Statistical experts come from GIZA++ and FastAlign posteriors trained on the same unlabeled parallel data used by the baselines. Training uses label budgets of 0, 50, 200 and 1,000 gold sentences. At zero labels, the weights are uniform and temperatures are fixed. Larger budgets apply only on GALE, where training data is plentiful.
The analysis has two parts. First, we read off the learned weights per language pair, to see which layers and priors matter, for example whether shallow layers gain weight for Ja-En and Zh-En. Second, we break errors down by token type: numbers, named entities, function words, and multi-subword words. This checks whether the orthographic gate specifically fixes rare-token errors, which DiscAlign and SimAlign leave unaddressed.
Baselines
The naive reference points are FastAlign and GIZA++. They need no labels and anchor the scale. The file reports De-En AER of 26.2 and 18.9 for these from the Cross-Align repository. Note that two GIZA++ Zh-En figures in the file disagree (19.4 in the repository and 35.1 in AWESOME). We will rerun GIZA++ ourselves under one fixed setup rather than quote either.
The zero-label neural reference is SimAlign's single best layer with Itermax extraction. It reports 18.8 AER on De-En and 21.6 on Zh-En per the Cross-Align repository. Our uniform-weight, zero-label variant must beat it to show that multi-layer fusion helps without any supervision.
The strongest existing unsupervised method is Cross-Align. It reports AER of 13.6 on De-En, 3.4 on En-Fr, 20.9 on Ro-En, 10.1 on Zh-En and 35.4 on Ja-En in Cross-Align. Awesome-Align is just behind at 15.6, 4.4, 23.0, 12.9 and 38.4. We use the released Cross-Align numbers and, where feasible, rerun it on our evaluation splits. We hold out gold sentences for our own supervision, so test sets shrink; all systems will be scored on the identical held-out portion. Beating it means lower AER on the same held-out sentences with at most 200 gold training sentences, while adding no cross-attention parameters.
On GALE, the strongest supervised competitor is DiscAlign, with F1 82.74 on English-Arabic and 73.41 on English-Chinese in DiscAlign. We retrain DiscAlign at each of our label budgets on the same backbone and data. Fusion counts as beating it if it matches DiscAlign's full-data F1 with fewer labels, or exceeds it at equal labels. We also replicate DiscAlign's English-to-Chinese NER projection experiment, on which it reports a 17.67 F1 boost at 500 sentences, as an extrinsic check.
The key ablation is early concatenation. The same raw expert maps are stacked as channels into a two-layer convolutional network with nonlinearities. It gets the same data, the same tuning budget of 30 trials per system, and comparable parameter count. This isolates where and how the evidence is mixed, since the inputs are identical.
Two further ablations remove per-expert bidirectional normalization (fusing raw scores) and remove the gate. They show which ingredient carries any gain.
Why not just…?
Why not just concatenate the maps and let a network learn the mixing? This is the most serious threat, and the parameter-saving argument from the vision papers does not apply. With roughly twenty input maps, a convolution over stacked channels is already tiny. The real difference is inductive bias: normalization before mixing and no nonlinearity before the sum. That bias should matter most at 50–200 labels and fade at GALE scale. The early-concatenation ablation, run at every label budget, settles this. If concatenation matches fusion at small budgets, we report that the log-linear bias is unnecessary and drop the claim.
Why not just use cross-attention as Cross-Align does? Cross-Align reports that its randomly initialized cross-attention layers need large parallel corpora, with its weakest relative standing on Ro-En. Speech fusion results in Attention Is Not Always the Answer also favor addition over cross-attention, though only for a binary task. Our Ro-En and Ja-En comparisons, at matched label and parallel-data budgets, test directly whether fusion closes the gap without the extra parameters. If Cross-Align still wins at full data, the honest conclusion is that fusion is the low-resource choice, not a replacement.
Why not just pick the best single layer, as SimAlign does? The best layer differs between SimAlign (layer 8) and Cross-Align (layer 11), and presumably across language pairs. Picking it requires labeled data anyway. Our learned weights will show whether the optimum is a sharp peak, in which case single-layer selection suffices, or genuinely spread across layers. The single-best-layer-with-tuned-threshold baseline at equal labels decides this.
Why not just average the experts post hoc with fixed weights? This is a real possibility, because our model is essentially a learned interpolation. The uniform-weight variant is exactly this baseline. If it matches the learned model, the contribution reduces to a finding: normalized multi-expert averaging is a strong, free aligner. That result would still be useful, and we would report it as such.
Decisive Checks
The proposal stands or falls on three results:
- Fusion's AER at 200 gold sentences, compared with the early-concatenation ablation and with Cross-Align on the same held-out De-En, Ro-En, Zh-En and Ja-En sentences.
- Fusion's GALE F1 curve over label budgets, compared with retrained DiscAlign.
- The rare-token error breakdown with and without the gate.
Gains concentrated at small budgets and on rare tokens would confirm the product-of-experts hypothesis. Uniform gains that persist at full data would suggest something else, most likely ensembling, is responsible. We would then report that instead.
References
- Log-Linear Models for Word Alignment
- Improved Discriminative Bilingual Word Alignment
- Discriminative Word Alignment with Conditional Random Fields
- SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings
- Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment
- A Discriminative Neural Model for Cross-Lingual Word Alignment
- Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion
- DenseSSM: State Space Models with Dense Hidden Connection for Efficient Large Language Models
- Word Alignment by Fine-tuning Embeddings on Parallel Corpora